Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective
Authors: Qishuai Wen
Chun-Guang Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Semantic Segmentation
Transformer
Decoder
Coding Rate
Principal Components
Primary area: interpretability_and_explainability
TL;DR: We derive a white-box decoder for Transformer-based semantic segmentation from the perspective of maximizing coding rate and computing principle components.

Abstract: State-of-the-art methods for Transformer-based semantic segmentation typically adopt Transformer decoders that are used to extract additional embeddings from image embeddings via cross-attention, refine either or both types of embeddings via self-attention, and project image embeddings onto the additional embeddings via dot-product. Despite their remarkable success, these empirical designs still lack theoretical justifications or interpretations, thus hindering potentially principled improvements. In this paper, we argue that there are fundamental connections between semantic segmentation and compression, especially between the Transformer decoders and Principal Component Analysis (PCA). From such a perspective, we derive a white-box, fully attentional DEcoder for PrIncipled semantiC segemenTation (DEPICT), with the interpretations as follows: 1) the self-attention operator refines image embeddings to construct an ideal principal subspace that aligns with the supervision and retains most information; 2) the cross-attention operator seeks to find a low-rank approximation of the refined image embeddings, which is expected to be a set of orthonormal bases of the principal subspace and corresponds to the predefined classes; 3) the dot-product operation yields compact representation for image embeddings as segmentation masks. Experiments conducted on dataset ADE20K find that DEPICT consistently outperforms its black-box counterpart, Segmenter, and it is light weight and more robust.

Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective
Qishuai Wen and Chun-Guang Li" School of Artificial Intelligence
Beijing University of Posts and Telecommunications, Beijing 100876, P.R. China {wqs, dbut. edu. cn
Abstract
State-of-the-art methods for Transformer-basse semantic segmentation typically adopt Transformer decoders that used to extract additional embeddings from image embeddings via cross-attention, refine either or both types of embeddings via self-attention, and project image embeddings onto the additional embeddings via dot-product. Despite their remarkable success, these empirical designs still lack theoretical justifications or interpretations. thus hindering potentially principled improvements. In this paper, we argue that there are fundamental connections be- tween semantic segmentation and compression, especially between the Transformer decoders and Principal Component Analysis (PCA). From such perspective, we derive white-box, fully attentional DEcoder for Principled semantiC segemenTa- tion (DEPICT), with the interpretations as follows: 1) the self-attention operator refines image embeddings to construct an ideal principal subspace that aligns with the supervision and retains most information; 2) the cross-attention operator seeks to find a low-rank approximation of the refined image embeddings, which is ex- pected to be set of orthonormal bases of the principal subspace and corresponds to the predefined classes; 3) the dot- dot-product operation yields compact representation for image embeddings segmentation masks. Experiments conducted dataset ADE20K find that DEPICT consistently outperforms its black-box counterpart, Segmenter, and it is light weight and more robust. Our code and models are available http:://gihbb.cco////saaaaaaaaaa
Introduction
Semantic segmentation has been fundamental task in computer vision for decades. In the supervised setting, the task aims to segment an image into regions corresponding to different predefined classes.
The dominant approaches for semantic segmentation have experienced significant shifts, in particular, from hand-crafted features [16] to deep learning, from Convolutional Neural Networks (CNNs) [24]
to Vision Transformers (ViT) 46. 35], and then from per-pixel classification to mask classification g
Recently, state-of-the-a methods 335 g 45 for Transformer baaemerr-baedd semantic segmentation
typically adopted the Transformer decoders inspired by DETR 2
Although they vary among different methods, the Transformer decoders typically consist of cross- attention operators that extract additional embeddings (known as class embeddings (35 45 or mask embeddings X ( from image embeddings, self-attention operators that refine either or both the additional embeddings and image embeddings, layer normalization (L.N) [21] and feedforward neural networks (FFN), which are the default compositions of a Transformer block [37], and one (or two for mask classification) dot-product operation of the two types of embeddings. Here, we illustrate in Figure two representative methods, i.c., Segmenter 35] and MaskFormer [
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

class
Ireage anis das urll
mragy enih:
transformer
decoder
FTN
nomm
b)
transformer decoder
self-atemtion
backhone
decoder
Borm
inage crlo
Figure 1: Illustration for Segmenter and MaskFormer. a) Segmenter. b) MaskFormer. e) Transformer block adopted by Segmenter. We omit the details of the Transformer decoder adopted by MaskFormer, which refines image embeddings and mask embeddings via self-attention respectively before the cross-attention operations.
Despite the empirical designs of the Transformer decoders being intuitive and having achieved remarkable success 7 (2), there still lack theoretical justifications or interpretations, thus hindering potentially principled improvement (such as identifying and addressing performance bottlenecks). We believe that the first step toward white-box models for Transformer-bass semantic segmentation to answer the following questions: 1) Why do the Transformer decoders outperform a position-wis MultiLayer Perceptron (MLP) that independently classifies each image embedding [35]? 2) What the underlying mechanism of the self- and cross-attention operators adopted by the Transformer decoders? 3) More importantly, is there principle for designing and improving the Transformer decoders?
In this paper, we argue that there are fundamental connections between semantic segmentation and compression, especially between the Transformer decoders and Principal Component Analysis (PCA), and that the principle of compression is all we need to derive white-box decoder akin to the Transformer decoder. To be specific, we extend the objectives PCA in the geometric and rank minimization views (38] to the context of the coding rate [I] 25 43 and, following the derivation in
CRATE (42, we derive self- and cross-attention operators for semantic segmentation by unrolling the optimization of these objectives.
Contributions. The contributions the paper are highlighted as follows.
We take further step along the fundamental connections between semantic segmentation and com- pression, by introducing PCA for understanding the empirical designs of decoders for Transformer- based semantic segmentation.
2. We extend the objectives well as the idea of PCA in terms of the coding rate to unrolled optimization, and thus derive a family of white-box fully attentional DEcoder for Principled semantiC segmenTation (DEPICT).
We conduct extensive experiments to evaluate the performance of our DEPICT and find that DEPICT consistently outperforms its black-box counterpart and shows desirable properties suggested by the derivation.
2 Related Work
2.1
Interpretability of Decoders for Semantic Segmentation
We cast decoders for semantic segmentation into four categories: 1) convolutional decoders
24 33
46] 2) MLP-based decoders [35] 6 26); 3) Transformer decoders [35] 45 4) clustering decoders 49 41 50
23 [7]. The core operations of convolutional decoders are convolution and pooling. Despite rooting signal processing,
interpretability of learned convolutional filters is limited. Among these decoders, segmentation masks are also features. And, as self- attention layer can express any convolutional layer [9, the self-attention operator we derive is very likely to be decomposed into convolutional filters. Due its powerful fitting ability, MLP-based

---

decoder is quite difficult to interpret. Nonetheless, single linear layer which is used by pre-tra pre-trained models [T5] 18 28 as the classifier can be viewed as a parametric softmax projection, which learns
prototype for each class 49
To knowledge, Segmenter 3 proposed the first Transformer decoder, which is used to perform
per-patch classification; whereas MaskFormer [8] formulated the task of mask classification, which performs clustering at first and then classifies the clusters. As a much broader concept, clustering decoders include all methods that are explicitly based on the idea of clustering. In particular, [I]] reformulates cross attention as a clustering process: whereas [17] exploits the principal directions for segmentation, based on the relationship between PCA and k-means [13] In this paper, we instead take compression perspective, which is more fundamental than clustering.
2.2 White-Box Models based on Coding Rate
The coding rate [ is an effective criterion for compression and is first introduced for segmentation by [25]. And a solid justification for the connections between image segmentation and compression can be traced back to 32), which argues that the optimal segmentation of an image is the one that will give the shortest coding length for encoding the image. Then, a principled framework, termed Maximal Coding Rate Reduction (MCR²) 43 was proposed to learn discriminative and diverse representations, in which MCR maximizes the difference between the coding rate of the ambient space and the sum of the coding rate for each class-speciff individual subspace and is shown to promote representations lie in a union of orthogonal subspaces.
By unrolling the gradient-based based optimization procedure of MCR2, a deep architecture, termed ReduNet 4 is derived as white-box counterpart the black-box of ResNets and CNNs, followed by CRATE 42 for ViTs [5] and CRATE-MAE [29] for masked autoencoders [18]. Intriguingly. CRATE shows a segmentation emergence similar to DINO B 44,, which is mainly attributed to the Multi- Subspace Self-Attention (MSSA) operator derived by [42]. These works have laid solid foundation for further investigation of the connections among compression, representation learning, and vision tasks at varying granularity. Especially, the role of normalization, e.g., ensuring the effectiveness of the coding rate, has been discussed in 1 4. As an analog, we adopt LayerNorm to normalize the scale of image embeddings before all attention operators that we derive.
3
Our Methods
3.1 Notations and Preliminaries
== Given arbitrary image for semantic segmentation, we Z (...... 2N] E DxN to denote set of image embeddings, where each image embedding 2; E R represents one of the N regular non-overlapping patches to which the image is split. Specifically, we assume that Z is zero mean, i.e., Z1 = O, where 1 E R is the vector of I's. For a Transformer decoder, we use Zo to denote the input, which is actually the output of the ViT backbone, and Ze is for Zo after being updated é times, where E < L, We use Q .q...... 4x]
E DD to denote the additional embeddings (or queries, or more precisely, cluster centroids), Qu for their initialization, Q, for Qo after being updated É times, where e < L2, and I for an identify matrix with proper dimension. We specially set K equal to the number of predefined classes, C, thus referring to the additional embeddings as class embeddings, or classifiers instead. A generalized case will be discussed in Appendix A.3
For PCA, what concern about are the leading C principal directions of Z and the associated C-dimensional principal subspace, which is denoted as S. For convenience, we simply refer to them as the principal directions and the principal subspace. And we use U [ui,.. ....... uc] DxC to denote an arbitrary set of orthonormal bases of S. Notably, we introduce PCA from two different perspectives here and will extend them the context of coding rate in the following sections. From geometric perspective, PCA minimizes the squared reconstruction error when recovering Z.i.c.,
min Z U. U.Y
U Ic.
(1)
From a rank minimization perspective, it seeks a low-rank approximation of Z, i.c.,
min| Z A
AlEE s.t. rank(A) = C.
(2)
3

---

Zo Z M Z Z M
Image Segmentation via PCA and DEPICT. Given an image, we segment it via PCA and DEPICT. We perform PCA its representations Zo and Z L,, respectively, where the first 10 principal directions are used as cluster centroids. We find that PCA can serve especially on the refined features, like L,,- We also observe that performing PCA on Zo is more likely to lead to an over-segmentation, which indicates that its principal subspace is not ideal.
Figure 2:
effective method image segmentation
Finally, introduce the concept of the coding rate of Z subject to certain distortion 0, which calculated as follows:
R(Z)+ 1
D
log det +
Zz").
(3)
Ne
3.2 Bridging the Transformer Decoders and PCA
For classification tasks, such as semantic segmentation, on the scale of an entire dataset, it is desirable have a Linear Discriminative Representation (LDR), which can be well modeled by a union of orthogonal subspaces 43 4
However, it is difficullt to explicitly model such a desirable structure when C' is relatively large, even at the cost of sacrificing the diversity each class (i.c., the dimensions of each subspace). Fortunately, we notice that Transformer decoder segments each image independently; that is, an embedding of one image would never attend to or interact with the embeddings from a different image. Additionally, despite that the intra-class variance or diversity is very rich for semantic segmentation on entire dataset, it would be limited within a single image. Therefore, focus
single image for now and then generalize it in Section 3.5
As shown in Figure the Transformer decoders for semantic segmentation typically project image embeddings onto additional embeddings to predict masks. In the case we focus on, it is projected onto the C-dimensional subspace spanned by class embeddings, where process of dimension reduction, i.e., compression, where the masks M Q Z E CCNN represent more compact features compared to Z. Intuitively, the subspace is expected to contain as much information as possible, which is crucial for capturing the rich intra-class variance for semantic segmentation. Mcanwhile, it is also desirable for the class embeddings to be orthonormal for more discriminative classification [19 30 43 [T]] In other words, seek to find a fow-dimensional subspace that best fits the image embeddings, which is exactly the idea behind PCA, and to find a set of orthonormal bases it as classifiers.
much smaller than D.
Therefore, we can view semantic segmentation
It is not difficult to show that the problem in I is equivalent to:
1
max
tU ZZ U) Var( Z), s.t. U = Ic, N
(4)
which indicates that the principal subspace should retain most variance (i.e., information). and the variances after projection onto the principal directions maximized. Therefore, contend that the principal subspace and the subspace spanned by class embeddings, as well as the principal directions and class embeddings, are fundamentally related; in ideal case, they equivalent. In Figure 2 we visualize several images segmented using PCA and find that PCA can indeed serve as an effective method, validating the above analysis.
In contrast to the Transformer decoders, a single linear layer can viewed as performing PCA on the entire dataset, at which scale the image embeddings definitely cannot be well fitted by C-dimensional subspace (i.e., performing bad compression). Meanwhile, as the intra-class variance in semantic segmentation is richer than in image classification, single static classifier (or prototype.

---

or more precisely, basis) is not sufficient to represent class well. This is why single linear layer clearly lags behind more complex methods in fine- seee-grrned classification tasks.
DEPICTCA
MscA
MSCA
a
P.
P
MSSA
MSSA
P
Figure 3: Illustration for DEPICT. Given an image for semantic segmentation, represent it as Zo by the ViT backbone. Segmenting it by performing PCA Zu. find that S of Zo is ideal. We thus adopt the MSSA operator refine image embeddings, iteratively constructing an ideal S. Performing PCA again on Z L, we find that the segmentation results are improved. Then, we adopt the MSCA operator to find a low-rank approximation of Zi, that lies in S as classifiers. For example, we use the dogs and cats on the right to represent image embeddings of two different classes in the feature space. Initially, the projections of dogs and cats onto S well linearly separable. DEPICT, however, constructs ideal S and effectively classify them.
3.3 Constructing an Ideal Principal Subspace via Self-Attention
Despite PCA being an effective method, which is also observed in [17], find that performing directly on Zo typically yields inferior segmentation results, such as oversegmentation, as shown in Figure 2 We attribute the reason for this an less ideal principal subspace Zo. which means that the relevant information for supervised semantic segmentation is either not contained in it or not significant enough. This hypothesis is intuitive since Zo expected be generic features and requires refinement to align with specific supervision. To this end, we propose to refine it to construct ideal principal subspace. As shown Figure the refinement can in fact alleviate the seer-eegmenttiion and improve the segmentation results.
Specifically, assume that P Pi..... Pel RRC is of orthonormal bases of ideal subspace. Now we optimize the objective of (4) with respect to (w.r.t.) to ensure that P is the principal subspace of Z after refinement, that is,
max
Varlp: Z);
(5)
where P is learned through backpropagation during training. However, instead of optimizing Z in directly, we choose maximize the projected coding rate onto these bases, that is,
max R(p. 2) R(P.z)== Σ log det(( + No ZZ P.).
This is because the coding rate is more generalized metric and is effective in high-dimensional spaces. Still, it is equivalent to (5 (as p zz P. is non-negative scalar). According to the seminal work 42). optimizing problem (6) via a gradient step with a learnable step size a > derives the MSSA operator, which in case be written
z z++ = (1 +
Ne MSSA(Z' P))
a
N
5

---

where the MSSA operator is defined as follows:
SSA (p))
MSSA(Z P)
N PL
Pcl
(8)
SSA Z Pc)
SSA(Z P.) (p: 2) softmax ((p. 2) (p. Z)) for C (I)...,C).
(9)
In 18
however. MSSA demands C' attention heads, each of which calculates an attention matrix in the shape of N N. As C' is always much larger than the number of heads of Multi-Heas Self-Attention (MHSA) 37 which is the black-box counterpart of the MSSA operator, thus the computation of (8) occupies an unacceptable amount of GPU memory.
To remedy this issue, we propose to maximize a lower bound of the sum of the projected coding rate onto the bases in groups, rather than maximizing them directly one by one. By our proof in Appendix D.I the sum of the projected coding rate onto a set of bases can be bounded below by the projected coding rate onto the subspace they span, i.c.
M
Σ
R(p'. Z) R(P" Z) 7,
(10)
where P' = p , P'll E RDxM consists of M different bases in P. and is product of and a constant w.r.t. Z. We thus divide the columns P into H C/M non-overlapping and maximize the lower bound for each group, respectively. We
groups, denoted P i..... P' thus reformulate the MSSA operator in our case as:
M
z = (1+a
z'
MSSA(Z P),
(11)
N
2e
SSA(Z P())
MSSA Z P) = M
[P 1 PH
(12)
Ne²
SSA(Z Pax)
SSA(Z Ph) = PP' Z) softmax
(P') T 2) (P) z)). for h E {1, H).
(13)
So far, we have derived self- self-attention operator which takes gradient step toward constructing an ideal principal subspace. In Appendix A.2 we discuss the differences among MHSA, the original MSSA, and our modified MSSA in (12). Additionally, our goal can also be achieved by minimizing the projected coding rate onto the bases of the orthogonal complement of the ideal principal subspace, with the only change being set a implementation.
Therefore, we do not constrain the sign of a in our
3.4 Finding a Low-Rank Approximation via Cross-Attention
With the ideal principal subspace learned via pockprroaagation and constructed via self-attention, the remaining problem is to find set of classifiers (i.c., class embeddings) for semantic segmentation. As discussed in Section 3.2, the principal directions would be desirable choice. However, shown in Figure PCA still yields inferior segmentation results compared to parametric and learnalle methods. We attribute this issue the principal directions not being flexible align with supervision, despite the fact that the constructed principal subspace has been. Meanwhile, being an unsupervised method, additional and challenging step that assigns each principal direction a predefined class. To this end, we derive an operator to extract a set of classifiers that satisfy the requirements as follows: 1) learns to align with supervision; span the principal subspace; 3) effectively abstract or represent the image embeddings.
PCA requires
For the last requirement, we propose to optimize objective that seeks to find a low-rank approxi- mation of Z in terms of the coding rate, i.c.,
min [R(Z) R (Q))
s.t. rank Q C.
(14)
where Q E DxN This is inspired by the rank minimization perspective of PCA in 2 and the objective of k-means which can be written as
min Z
2 s.t. rank(V) = C,
(15)

---

where the columns of V E RDNN consist of cluster centroids. Using the coding rate to measure the approximation, the objective (14 is more generalized than ( and (1)). In Appendix D.2 we prove that both k-means cluster centroids and the principal directions are reasonably good (or even the optimal) solutions under certain conditions, for (14).
Intuitively, since that the columns of Q span subspace with which dimension is lower than that of Z, we have R( ((Z)= 2 R(Q), which will be validated by the experiments in the Appendix C Thus, we equivalently maximize the coding rate of Q and derive that:
Q -+1 = (1+
D Ne
q al
D '²Q softmax Nea
(16)
which is similar to (2) Note that compared to using the Frobenius norm in 2 and 15 the
approximation in (14) terms of the coding rate is relatively loose due its invariant property 4
thus optimizing over Q turns out to be irrelevant to Z. Therefore, replace some in 16 with Z
to further encourage Q to approximate Z, and thus we have:
D ++1 = -1+ Ne q -a al Ne
12 (2)
D
softmax 'z' Q²).
(17)
Rather than Q which is redundant, what we are indeed concerned with is Q. Thus, we simplify (77 by updating each y, only once, i.e.,
D TotNN22 N2 -
D -al NZ) 2zz' softmax (z Q²).
= (1+ -
(18)
For the first requirement, we set Qo by learnable parameters; in other words, the alignment to predefined classes is learned by adjusting the starting point of gradient optimization. For the second requirement, we confine the updates of Q to the principal subspace of Z by adding projections onto P. Similarly to 122 we reformulate
(8)
M
M
Q++1= = (1 11+0
N
Q
a
Ne
MS' Z. P.P),
(19)
SCA(Q Z,P
MS Z, P) = M
[P
P'r]l
(20)
N
SCA(Q 2,P'))
SCA (Q) P') = (PA Z) softmax (PP) (P', T Q)):
for h E (1,.... ........)) H), (21)
which we refer to as Multihead Subspace Cross-Attention (MSCA).
3.5 Decoder for Principled Semantic Segmentation
From single image to the entire dataset, we should consider multiple "principal subspaces" 38 and thus allow P to model more bases by raising the number of columns of P from to a hyperparameter K that is larger than C. We expect that the principal subspace, as well as the class embeddings, of different images vary, but the class embeddings of the same class lie in a low-dimensional subspace, and all the subspaces of class are orthogonal, thus satisfying our anticipation for LDR on the entire dataset.
we have fully attentional white-box decoder as shown in Figure which iteratively constructs an ideal principal subspace by refining image embeddings via the self-attention operators, and then find low-rank approximation of the refined image embeddings that lies in the principal subspace and corresponds to predefined classes via the cross-attention operators. As our derivations demonstrate that the principle of compression is all we need for designing the decoders, we refer to our approach as the DEcoder for PrIncipled semantiC segmenTation (DEPICT), and our approach described above that extracts additional embeddings via cross-attention is referred to as DEPICT-CA.
By stacking and combining the two steps of Section 3.3 and 3.4
We note that the additional embeddings can also be extracted by concatenating the two types embeddings and updating them simultaneously via self-attention alone, as shown in Figure [ a). In Appendix [D.3] we prove that it implicitly performs cross-attention, thus it can be interpreted by our derívations in Section 3.4 We implement a simpler variant of DEPICT that extracts the additional embeddings via self-attention and refer to it as DEPICT-SA.
[15 35 12]: that is,
7

---

Experiments
4.1 Segmentation Performance
Baselines and implementation details. To evaluate the segmentation performance of our DEPICT, we conduct extensive experiments on datasets ADE20K [47], Cityscapes [10], and Pascal Context datasets 2 compared to Segmenter and MaskFormer. As shown in Figures and 3 Segmenter serves as the black-box counterpart to DEPICT, while MaskFormer is more advanced method that adopts pixel decoder, mask classification formulation, and hierarchical backbones and advanced loss functions. For fair comparisons to Segmenter, we use the same settings, including backbones, data augmentation, optimization and inference. We humbly refer readers to [35 for more details. On ADE20K, we use three MSSA layers for DEPICT-SA and three MSSA layers followed by three MSCA layers for DEPICT-CA other two datasets can be found in Appendix
most variants, whereas the exceptions and more settings the
Table I:
Comparison ADE20K validation set. We compare DEPICT with Segmenter and MaskFormer.
The best-performing result DEPICT is highlighted.
Model
Backbone
Resolution
mloU(ss/ms)
#params
FLOPs
VIT-T
512x512
38.1/38.8
1M
2.2G
ViT-S
512x512
45.3/46.9
4M
6.7G
Segmenter
VIT-B
512x512
48.5/50.0
16M
22.3G
VIT-L
640x640
51.8/53.6
28M
60.4G
MaskFormer
Swin-L
640x640
54.1/55.6
15M
47.7G
VIT-T
512x512
38,4/39.4 4
0.2M
1.4G
ViT-S
512x512
45.8/47.0
0.6M
3G
DEPICT-CA
VIT-B
512x512
49.0/50.0
1M
4G
VIT-L
640x640
52.8/53.3
2.5M
16.5G
VIT-T
512x512
39.3/40.7
0.2M
2.9G
ViT-S
512x512
46.7/47.7
0.4M
3.4G
DEPICT-SA
VIT-B
512x512
49.2/50.7
0.8M
4.2G
512x512
52.5/54.0
2M
9.5G
VIT-L
640x640
52.9/54.3
2M
17.8G
Table 2: Comparison on the validation sets of Cityscapes and Pascal Context, All are based on VIT-L.
Cityscapes #params
PascalContext #params
Model
mloU(ss/ms))
FLOPs
mloU(ss/mss)
FLOPs
Segmenter DEPICT-SA
79.1/81 1/81./1.
15.8M
45.2G
1/59.0 57.9/58.6
28 28.4M
30.0G
78.8/81.0
0.1M
2.4G
1.6M
5.6G
Table 3: Comparison subset of ADE20K training set. All based on VIT-S.
Dataset Size
Model
45.0
4K
12K
16K
20K
Segmenter DEPICT-SA
38.31
41.87
43.42
44.61
45.37
40.0
Segmenter
36.42
41.75
43.34
45.12
46.72
DEPICT-SA
DEPICT-CA
35.68
41.11
42.33
44.61
45.76
37.5
DEPICT-CA
12
16
20
Size(K)
Performance comparisons and analysis. From Table [ we read that all variants DEPICT outperform Segmenter with significantly fewer parameters and FLOPS on ADE20K validation set. In

---

Segmenter
DEPICT-SA
DEPICT-CA
Segmenter
DEPICT-SA
DEPICT-CA
Figure 4: Investigating orthogonality in DEPICT. Left: PTP; P: Right: Q Q. All variants based
ViT-L. Since that the MHSA operator contains three parameter matrices, unlike MSSA which has only one, we visualize the matrix responsible for transforming queries. Notably, all the Q's are normalized, whereas is not.
DEPICT-SA
DEPICT-CA
Figure 5: Inner product of class embeddings across images. We group the class embeddings by their classes and visualize the inner-produce among them. We exemplify 30 classes across 100 images. All variants are based VIT-L.
particular, the best-performing variant, DEPICT-SA based on ViT-Large with an input resolution of 640x640, surpasses Segmenter by 1.100.7 mloU for single/multi-scale inference, while using only 1/14 of the parameters and 1/3 of the FLOPs. We mainly attribute the efficiency DEPICT to its removal of the FFN block, which plays no role in interpretations. Such redundancy has been observed empirically in the field of NLP [1]. but our ablation study in Appendix C shows that naively adopting MSSA while removing the FFN from Segmenter results in worse performance. Moreover, [4] proves that pure attention causes the rank of tokens to decrease rapidly with depth, which is desirable for
construct ideal principal subspace. In particular, according [35) both finer segmentation granularity and more performant backbone can significantly boost segmentation performance, partly explaining the inferiority of DEPICT compared MaskFormer. In Table we find that PICT performs slightly worse than Segmenter. We believe that this due to the limited size of the dataset; the training set sizes of ADE20K, Cityscapes, and Pascal Context about 20K, 3K and 5K, respectively. As Table shows, DEPICT requires a sufficient amount of data to demonstrate its superiority.
4.2 Desirable Properties of DEPICT
Orthogonal properties. According to our interpretations, the parameter matrices of attention are consisted of orthonormal bases, and the extracted class embeddings are fundamentally related to the principal directions, which are also orthonormal. As shown in Figure 4 both P and Q are very close to being orthonormal in DEPICT. Although the parameter vectors learned by Segmenter are also nearly orthogonal, their norms too small. Despite there semantic similarities among the predefined classes, the class embeddings of Segmenter are excessively related. Moreover, as we
Projected
Pro aetted
3.44
340
0.5
300
Layernsa
the parameter matrix of its first head, say Pi and measure the mean projected coding rate of image embeddings onto the subspace spanned by fL Each polyline reflects the changes of the projected coding rate onto a particular subspace and the layer index of the subspace is indicated by a vertical dash line in the same color. Left: R(p, T Z), across layers. Right: R(P p 2)/re 2)),
Figure 6: Measuring the coding rate layers. Given the 2-th layer,
2), across layers.

---

expected in Section 3.5 and shown in Figure 5 the class embeddings of different classes nearly lie in a union of orthogonal subspaces, and thus the image embeddings are very likely to satisfy assumption for L.R..
Table 4: Investigating robustness of DEPICT under parameter perturbation. We experiment with four types of perturbations: 1) the P of each attention operator undergoes random orthogonal transformation, OK B 2) the P/, of each head undergoes random orthogonal transformation, Our IR RRMMMM,35 orthogonalizing Pi; 4) adding random noise from a Gaussian distribution with zero mean and variance for each parameter independendly. The baseline is Segmenter, and we mark the improvements in the table.
Backbone ViT-B
DEPICT-SA
ViT-S
VIT-L
DEPICI-CA
P, POR k Ortho(P))
46.7(+1.4)
49.2(+0.7)
52.9(+1.1)
42.A(+27.8)
47.3(446.1)
52.0(+51.4)
DEPICT-SA
42.6(+27.6)
45.7(144.5)
51.8(+51.2)
Segmenter
0.00
0.11
0.02
0.3
0.04
0.05
0.10
Measuring the coding rate and robustness. On the ViT-L variant of DEPICT-SA with an input resolution of 512x512, we measure the projected coding rate of image embeddings onto the subspaces spanned by Pa, across layers, shown in Figure 6. We find that the sign of the learned step size distinguishes two types of subspaces (distinguished by red and blue regions in the figure): one onto which the projected coding rate increases and the other onto which it decreases, which is consistent with our derivations. Furthermore, we measure the robustness of DEPICT under four types of parameter perturbations and find that DEPICT is surprisingly robust whereas Segmenter collapses, as shown in Table We attribute the robustness of DEPICT to its awareness of and modeling for low-dimensional subspaces, which are not significantly altered under the parameter perturbations. Actually, the seminal work [36] has already noticed that it is the subspace, rather than individual units, that contains the semantic information in the high layers of neural networks. Our work demonstrates that this intriguing property can be strengthened by improving model designs, which yields better robustness.
5 Conclusion and Future Work
We proposed a compression perspective view the Transformer decoders widely adopted in Transformer-based semantic segmentation, where we expect that class embeddings are actually the principal directions and thus they span the principal subspace and segment images via PCA. In experiments, we found that the principal subspace of the generic features extracted by the encoder not ideal and that the principal directions are not flexible enough align well with predefined classes. To this end, we extended the objectives of PCA to construct an ideal principal subspace and find low-rank approximation of image embeddings classifiers. By unrolling the optimization procedure of these objectives, we derived family of fully attentional white-box decoders, called DEPICT, providing theoretical interpretations for the empirical designs the Transformer decoders. Experiments conducted ADE20K have shown that DEPICT consistently outperforms its black-box counterpart, Segmenter, using significantly fewer parameters and FLOPs. We further validated the effectiveness of DEPICT Cityscapes and Pascal Context datasets and investigated that DEPICT possesses desirable properties, such as orthogonality and robustness, as we expected and derived.
We believe that our work serves as promising first step toward developing comprehensive interpretation framework for Transformer- based semantic segmentation, and further efforts are needed to contribute to this goal. Focusing on interpretability, we use relatively simple implementation for DEPICT from architecture designing to training tricks, compared to state-of-the-art methods. Whether the improvements for black-box models, such as hierarchical transformer encoders, pixel decoders, the mask classification formulation, and masked attention, are compatible with DEPICT remains an open question to explore.
10

---

Acknowledgments and Disclosure of Funding
The authors would like to thank the constructive comments from anonymous reviewers. This work was partially supported by the National Natural Science Foundation of China under Grant 61876022.
C.-G. Li is the corresponding author.
References
[1] Shaked Brody, Uri Alon, and Eran Yahav. On the expressivity role of layemorm in transformers' attention. In ACI. (Findings), pages 14211-1422) Association for Computational Linguistics, 2023.
[2] Nicolas Carion, Francisco Massa, Gabricl Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV (I), volume 12346 Lecture Notes in Computer Science, pages 213-229. Springer, 2020.
[3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Rojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, pages 9630-9640, IEFF, 2021.
[4] Kwan Ho Ryan Chan, Yaodong Yu, Chong You, Haozhi Qi, John Wright, and Yi Ma. Redunct: A white-box deep network from the principle of maximizing reduction. J. Mach. Learn. Res., 23:114:11114103, 2022.
[5] Liang-Chich Chen, George Papandreou, Jasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplah: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE Trans. Pattern Anal. Mach. Intell., (0(4)):344-888 2018.
[6] Shoufa Chen, Enze Xie, Chongjian Ge, Runjian Chen, Ding Liang, and Ping Luo. Cyclemlp: A mip-like architecture for dense visual predictions. IEEE Trans, Pattern Anal. Mach. Intell., 45112)114884114300, 2023.
[7] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, pages 1280-1289. IEEF, 2022.
[8] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov, Per-pixel classification is not all you need for semantic segmentation. In NewrlPS, pages 17864-17875, 2021.
[9] Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self-attention and convolutional layers. In ICLRR OpenReview.net, 2020.
[10] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bertt Schicle. The cityscapes dataset for semantic urban seene understanding. In CVPR, pages 3213-3223. IEEE Computer Society, 2016.
[11] Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley, 2001.
[12] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In ICLR. OpenRevicw.net, 2024.
[13] Chris H. Q. Ding and Xiaofeng He. K-means clustering via principal component analysis. In ICML, volume of ACM International Conference Proceeding Series. ACM, 2004.
[14] Yihe Dong. Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: pure attention loses rank doubly exponentially with depth. In ICML, volume 139 Proceedings of Machine Learning Research, pages 2793-2801 PMLR, 2021.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition scale. In ICLR. OpenReview.net, 2021.
[16] Clément Farabet, Camille Coupric, Laurent Najman, and Yann LeCun. Learning hicrarchical features for scene labeling. IEEE Trans. Pattern Anal. Mach. Intell,, 35(8): 1915-1999999991 1929, 2013.
[17] Oliver Hahn, Nikita Araslanov, Simone Schaub-Meyer Meyer, and Stefan Roth. Boosting unsupervised semantic segmentation with principal mask proposals. CoRR, abs/2404. 16818, 2024.

---

[18] Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 15979-15988. IEEE, 2022.
[19] Elad Hoffer, Itay Hubara, and Daniel Soudry. Fix your classifier: the marginal value of training the last weight layer. In ICLR (Poster). OpenReview.net, 2018.
[20] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. In ICCV, pages 3992 4003. IEEE, 2023.
[21] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. ArXiv e-prints, pages arXiv-1607, 2016.
[22] Xiangtai Li, Henghui Ding, Haobo Yuan, Wenwei Zhang, Jiangmiao Pang, Guangliang Cheng. Kai Chen, Ziwei Liu, and Chen Change Loy. Transformer- based visual segmentation: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[23] James Liang, Yiming Cui, Qifan Wang, Tong Geng, Wenguan Wang, and Dongfang Liu. Clusterfomer: Clustering as A universal visual learner. In NewrIPS,
[24] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In CVPR, pages 3431-3440. IEEE Computer Society, 2015.
[25] Yi Ma, Harm Derksen, Wei Hong, and John Wright. Segmentation of multivariate mixed data via lossy data coding and compression. IEEE Trans, Pattern Anal. Mach. Intell,, 1999):546-1566 2007.
[26] Yizhe Ma, Fangjian Lin, Sitong Wu, Shengwei Tian, and Long Yu. Prseg: A lightweight patch rotate MIP decoder for semantic segmentation. IEFE Trans. Circuits Syst. Video Technol., 33(1 11:6860-6871, 2023.
[27] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Scong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR, pages 891-898. IEEE Computer Society, 2014.
[28] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran. Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. Trans. Mach. Learn Res., 2024, 2024.
[29] Druv Pai, Sam Buchanan, Ziyang Wu, Yaodong Yu, and Yi Ma. Masked completion via structured diffuusion with white-box transformers. In ICLR. OpenReview.net, 2024.
[30] Vardan Papyan, XY Han, and David I. Donoho. Prevalence neural collapse during the terminal phase of deep lcarning training. Proceedings of the National Academy of Sciences, 1 (7744))24652-24666 2020.
[31] Telmo Pires, António Vilarinho Lopes. Yannick Assogba, and Hendra Setiawan. One wide feedforward all you need. In WMT, pages 1031-1044. Association for Computational Linguistics, 2023.
[32] Shankar R. Rao, Hossein Mobahi, Allen Y. Yang. Shankar Sastry, and Yi Ma. Natural image segmentation with adaptive texture and boundary encoding. In ACCV (1), volume 5994 of Lecture Notes in Computer Science, pages 135-146. Springer, 2009.
[33] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI (3), volume 9351 of Lecture Notes in Computer Science, pages 234-241. Springer, 2015.
[34] Axel Ruhe. Perturbation bounds for means of eigenvalues and invariant subspaces. BIT Numerical Mathematics, 1033):33-334, 1970.
[35] Robin Strudel, Ricardo Gareia Pinel, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, pages 7242-7252. IEEF, 2021.
[36] Christian Szegedy. Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR (Poster), 2014.
[37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017.
[38] René Vidal, Yi Ma. and S. Shankar Sastry. Generalized Principal Component Analysis, volume 40 of Interdisciplimary applied mathematics. Springer, 2016.
12

---

[39] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, José M. Álvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurlPS, pages 12077-12090. 2021.
[40] Jinrui Yang, Xianhang Li, Druv Pai, Yuyin Zhou, Yi Ma. Yaodong Yu, and Cihang Xie. Scaling white-box transformers for vision. arXiv preprint arXiv:2405.20299 2024.
[41] Qihang Yu, Huiyu Wang. Siyuan Qiao, Maxwell D. Collins, Yukun Zhu, Hartwig Adam, Alan L. Yuille, and Liang- Chich Chen. k-means mask transformer. In ECCV (29), volume 13689 of Lecture Notes in Computer Science, pages 0888-07. Springer, 2022.
[42] Yaodong Yu, Sam Buchanan, Druv Tianzhe Chu, Ziyang Wu, Shengbang Tong, Benjamin D. Haeffele. and Yi Ma. White-box transformers via sparse rate reduction. In NeurlPS, 2023.
[43] Yaodong Yu, Kwan Ho Ryan Chan, Chong You. Chaobing Song, Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. In NeurlPS, 2020.
[44] Yaodong Yu, Tianzhe Chu, Shengbang Tong, Ziyang Wu, Druv Pai, Sam Buchanan, and Emergence of segmentation with minimalistic white-box transformers. arXiv preprint arXiv:2308. 16271, 2023.
[45] Bowen Zhang, Zhi Tian, Quan Tang. Xiangxiang Chu, Xiaolin Wei. Chunhua Shen, and Yifan Liu. Segvit: Semantic segmentation with plain vision transformers. In NewrIPS, 2022.
[46] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H. S. Torr, and Li Zhang. Rethinking semantic segmentation from a sequence-to- sequence perspective with transformers. In CVPR, pages 6881-6899 Computer Vision Foundation / IEEE, 2021.
[47] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. Int. J. Comput. Vis., 127(3):302 321, 2019.
[48] Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Animashree Anandkumar, Jiashi Feng, and José M. Álvarez. Understanding the robustness in vision transformers. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 27378-281 27394, PMLR, 2022.
[49] Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Lue Van Gool. Rethinking semantic segmentation:
A prototype view. In CVPR, pages 2572-2583, IEEE, 2022.
[50] Alex Zihao Zhu, Jieru Mei, Siyuan Qiao, Hang Yan, Yukun Zhu, Liang-Chied Chen, and Hearik Kret- zschmar. Superpixel transformers for efficient semantic segmentation. In IROS, pages 7651-7658, 2023,
[51] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. In NeurlPS, pages 29820-29834, 2021.
13

---

Appendix
A Further Discussion on DEPICT
In this section, we aim to reiterate points that may lead to confusion, including subspaces and attention operators. Then we move to discuss the prospects and challenges for generalizing DEPICT to the mask classification formulation.
A.1 Reiterating All Mentioned Subspaces
Before diving into the MSSA operator, would like clarify several subspaces mentioned in work. We referenced "a union of orthogonal subspaces" and its connections to L.DR and MCR in Section and single image, the extracted class embeddings will lie within a union of orthogonal subspaces on the scale of the entire dataset, and observed it indeed occurs in Section 4.2 Each of these subspaces corresponds to represents predefined class in our work and 43
and Section 3.2 Then we expected in Sc.tion that, although our derivations target
We argue that the principal subspace of Zo is not ideal and assume ideal principal subspace spanned by P in Section 3.3 Thus, there are two types of principal subspace as image embeddings refined: the current principal subspace of Ze and the ideal principal subspace we desire, until they converge on Z -,,- Moreover, in Section 3.5 we mentioned "multiple principal subspaces" on the scale of the entire dataset, indicating that each image corresponds a distinct principal subspace.
In Section 3.3 we propose to maximize the projected coding rate onto Ps. which consists of bases of the ideal principal subspace. As generalize the derivation to the entire dataset scale in Pk. actually consists of M of bases of the "multiple principal subspaces".
Section 3.5
A.2 Differences among Self-Attention Operators
We discuss the differences among three types of self-attention operators: MHSA, the original MSSA, and our modified MSSA. There are four parameter matrices in MHSA. Nevertheless, due to parameter sharing, our MSSA has only one parameter matrix, P. and a learnable scalar, a, as the step size. The original MSSA, however, adopts an additional learnable parameter matrix for For image classification, MSSA-based models lag slightly behind semantic segmentation. Following the idea of MCR despite decoupling its heads from classes, the original MSSA still models homogeneous subspace in each head; in other words, these heads still correspond to some more basic or finer-grained concepts than classes. However, each head of our MSSA models heterogeneous subspace, since its bases are likely related to several different classes.
simplicity of implementation
MHSA 42 40
whereas our work demonstrates its superiority for Transformer-
A.3 Generalizing to Mask Classification
For simplicity of analysis and implementation, our work focus on class embeddings, rather than the more general case of mask instance embeddings proposed by (8) This couples each class with merely one principal direction within an image. However, an image typically contains a small part of all predefined classes and each class allows rich intra-class variance, thus requiring more than one principal directions to represent itself. Therefore, it is rather natural to generalize to mask classification the basis of our interpretations. Moreover, is desirable segment images finer granularity than patches-dow to the pixel level-by similarly adopting a pixel decoder. However, this requires interactions among features different scales, whereas our current work is limited to non non-hierarchical features.
B Implementation Details
Implementations of derived operators. As we perform Layer Normalization (LN), which is used to normalize representations and then learn to scale them, before all attention operations DEPICT, the scaling operations within derived operators can be simplified. Taking MSSA as example,
14

---

similarly for MSCA, it is implemented as follows:
g =z a MSSS(Z e P):
(22)
SSA | P()
( P) P PH
(23)
SSA(Z PHH) (PP (P) T 2)), for fl....,H. H).
SZ PA, (P', Z softmax
(24)
where a is actually a scaled step size.
Variant settings. For DEPICT-SA on the ADE20K dataset, we used three MSSA layers and set #heads dii head to 3 100 across all variants, with the exception six MSSA layers for the ViT-L variant. For DEPICT-CA, we used three MSSA layers followed by three MSCA layers, and set #heads dim head to x 100 in MSSA and #heads head to x550 in MSCA, across all variants, with the exception of setting x in MSSA for the ViT-S variant and six MSSA layers for the ViT-L variant. Only ViT-L variants of DEPICT-SA are evaluated on Cityscapes and Pascal Context, using two and four MSSA layers, and setting xheadsxding head 4 19 and 4x60, respectively.
C Additional Experiments
Investigating the relative relationship between RZZ) and RQ).. To make the visualization more intuitive, we measure R(Q). which is larger than (Q) since that an image contains only small part of all the classes. Thus, R(Q) serves as an upper bound of R(Q)). As shown in Figure 7 R(Q) is indeed smaller than R(Z) and the difference between them roughly decreases as the depth increases.
relatively small number of heads produces the best performance, implying that an over-tight lower bound may make training less effective. In [48] it is also observed that the performance initially increases with the number of heads, then declines.
Ablation studies. As shown in Table 5
Visualization of more variants. We present the visualization of the inner product of the parameter
vectors and class embeddings for all variants trained on ADE20K in Figures and TO We also provide additional segmentation results via PCA and compare them to those from DEPICT in Figure
g
DEPICT-SA
DEPICT-CA
R12)
H10)
H(q)
X20
DE
050
200
900
900
Indera
index
Figure 7: Measuring R(Z) and R(QQ) across layers. All are based on ViT-L with an input resolution of 640x640.
D Relevant Proofs
D.1 Lower and Upper Bounds for the Projected Coding Rate
E DDN and an M -ddieensional subspace spanned by a set of orthonor- mal bases P' E IR af af the projected coding rate of Z onto p' is:
Given image embeddings
M
R(P"T Z: c) log (I +
(P' (P T)
(25)
Ne²
Specifically, the projected coding rate onto one of the bases of the subspace, say p/. is
1
R(p." /T Z:c) 2 log det(1 + Ne (p. Z),,' T Z)
(26)
15

---

Table 5: Ablation studies. Left: All hased VYT-$ and compared the naive implementation which adopts MSSA and ISTA, with setting #heads ddim head to 3x50, #layer dropout to 0.0. Right: Investigating the impact of #heads and dim head.
variants
mloU
#params
#head x dim head
mloU
MSSA + ISTA
45.6
0.47M
VIT-S Backbone
#heads
45.4(-0.2)
0.47M
1x300
45.0
#heads =
45.6(+0.0)
0.47M
2x 2x150
45.7
#layers 1
45.2(-0.4)
0.27M
3x
46.7
#layers = 3
45,8(+0.2)
0.68M
4x75
46.3
dropout 0.1
45.2(-0.4)
0.47M
1x384
46.2
MSSA only
45.4(-0.2)
0.18M
VIT-B Backbone
ISTA only
44.9(-0.7)
0.36M
3x50
48.8
MSSA + MLP
46.2(+0.6)
2.5M
3x 100
49.3
MHSA ISTA
44.9(-0.7)
1.5M
3x150
49.4
MHSA + MLP (Segmenter)
45.3(-0.3)
4.1M
3x200
49,4
DEPICT-SA
46.7(+1.1)
0.41M
3x250
49.5
DEPICT
PCA
DEPICT
PCA
Figure 8: Image segmentation via PCA and DEPICT. Given an image, we segment it via DEPICT and PCA, respectively. We perform PCA on the representations Zi, and use the first 10 principal directions cluster centroids. We find that, with ideal principal subspace constructed, PCA method image segmentation, and captures additional information compared DEPICT, which is trained an end-to-end end manner. For example, numbering the images in subfigure row-wise from a to f. differentiates two different walls (see subfigure a); tombstones and grass (see subfigure c); sitting and standing persons (see subfigure f); sky, cloud, and power lines (see subtigure g). However, unsupervised method, PCA still does align well with supervision, especially in complicated scenarios (see subfigure b and d).
surprisingly effective
16

---

Figure 9: Visualization of the inner product of parameter vectors. From top to bottom, each row is based Segmenter, DEPICT-SA, and DEPICT-CA, respectively; from left to right, each column is based on ViT-T, VIT-S, ViT-B, and ViT-L, respectively.
Figure 10: Visualization of the inner product of class embeddings. From top bottom, each row is based on Segmenter, DEPICT-SA, and DEPICT-CA, respectively; from left right, each column based on ViT-T, ViT-S, ViT-B, and VIT-L, respectively.
17

---

By Lemma A.2 of 43 the projected coding rate in Eq. (25 can also be calculated as
R R(P"T e; = log del (T + Ne
(PT 2) (P"T 2)),
(27)
which is equivalent to
M z' P'p"T Z)
log det(IM +
(28)
Ne
p'1
M
log de
z 1p......P'' vl
:
Z)
(29)
Ne²
p T
M
M
log del
+
ZT
Ep'pi
Z)
(30)
2
Ne
1
1
M Ne
p.pi' z 2)
-
log det
Σ
I
z
(31)
M
By Lemma A.I of 43 the right-hand side of is bounded below by
M
M Ne²
log det((N)) M +
Z ''pp'' 2).
(32)
2M
In other words, we have shown that
IT
Z
R(P Z:e)
R
(c)
Rlp Z: Me).
(33)
M
M
M
By the proof of Lemma A.4 in 45 have that
log det (X) < log det Y) + trace (race((-lx)) - MN, for all {X, C S
(34)
where SMN denotes the set of positive symmetric matrices of size MN x MN. We take the specific values for and Y as follows:
Z p' 1p' Z
0
Z
P2 P2
Y I MN + Ne²
(35)
0
O
z T p' MD' M z
T
Z
(2
P;
'D:
(T
0
M 2
0 0
X MN
(36)
By the property of determinant for block diagonal matrix, we have
M
1
log det(det
log det +
Ne Ne=(D. (p,
"Z(",))p,"" TZ)T),
(37)
M zT
log det log det IN
TT(
ET P/P;"' (2))
(38)
No
Moreover, we have that
(II+ TT
M zT
(
[Ep'D,")
M
2 2))
trace (Y X) = trace
IN
Ne
M
((I+
zJ P.P.." Tz 2 )
trace
(39)
Ne²
2
18

---

Without loss of generality, we assume that
( +
1 Ne NoZ 2'P'z -
/ trace
)
( N22 z piP: Tz
Strace
for i M,,.
(40)
Ne
then the quantity in 39 is upper bounded by
M
M Z Zz
( P.P. (2) + (M - 1)8
trace
(IN+ + Ne
Z p' p'pp' 2IN
2e
(41)
M
=trace
(IN+ Ne
p' T Z
1
D)
(M Σ IN +
1
Z p.P. 1T Z - (M2 1 - 1)IN)
Ne
1=1
+(M - 1)8
(42)
=M + M
Σ
trace
(IN+
1
Z'
Z
p'p'' - Z)
-) (IN +
Z p'p'p. (2P - MMM 1)3.
Ne²
Ne
2
(43)
Without loss of generality, we assume that
trace
(IN +
Ne z P'P''''
2)
+
No
1
z p'p' (u
1
staace
(IN + Ne
z P' 2)
(IN +
1T 2) Z p.P:
i (M)
(44)
Ne
then
43 is upper bounded by
(IN+ 1 x 1''p'pp''2 2) 1( No22 t D' MP MP'' 2)
1
MN MMM 1)trace
(45)
Ne
(p' (Ux Ne (2' ( 2)) N₂² (v' 2) (v' 21)
-MN + MMM 1)trace
(46)
Since that the rank-1 1 matrix (p', T)p(p' Z) E NNN has a single non-zero eigenvalue, i.e.,
NVr((p') Z) (p', (p' T) By Ruhe's trace inequality 34 bounded by
we have that 46 is upper
MN + M(M 1) Var(p' T Z)
(47)
Varlp' _ () + T
Var(p' M Z)
SMN + M(M 1)
(48)
Var(p' ( Z)
We argue that since p' lies in the principal subspace in our case, the projected variance on different
bases does not differ significantly; in other words, the ratio listed above is upper bounded by constant with respect to Z. We therefore use to denote the product of M - 1) and this
constant. Now, by using (37
388 488) and 33 we derive
R(P"T Z;6) < Σ R(p 2:0)
(49)
In summary, we have proved that
M
M R(D T Z; Mc) S R(P'T Z:e) < Σ
R(p Z;e) +
(50)
M
i=1
19

---

D.2 Evaluate the Quality of Solutions for Low-Rank Approximation
In this section, we evaluate the quality of the solutions, specifically the principal directions and the cluster centroids k-means, for the low-rank approximation problem measured by the coding rate defined in Eq. 14). The coding rate of and Q given by
1 log d D +
D
R(Z)
zz Ne
(51)
2
D
R(Q) log det(ID)
QQ
(52)
Ne
Specifically, we assume that Q consists of the leading C principal directions.
Evaluating principal directions. Letting there be n, instances of q; in Q, where Σ1 n = N, we have
QQ
(53)
1=1
Qdiag mi :1.... nc )
(54)
(Qdiag (n) n (Qdiag(n) ..... nel).
(55)
By Lemma A.2 of 43
when calculating the coding rate of Q, we can rewrite (55) as follows:
(Qdiag(n ( ((ddia
(nf nt
(56)
=diag [ I Qdiag(ni )
(57)
Since that the principal directions are orthonormal, Eq. (57) turns out be diag(n)...... nc). By using PCA, the principal directions are essentially the eigenvector ZZ i.c.,
zz Q = Qdiag( (ddaggAA.....Ach.
(58)
where d
Ào is the leading C cigenvalues of ZZT. Therefore, long each n; is sufficiently close to A, E {l....,C. C). the principal directions are a sufficiently good solution. By taking transpose both sides of (58) and multiplying by Q, get
Q zz Q ... TTQQQQQQ (di..().....Ac...
(59)
We find that X; is actually the variance projected onto q: multiplied by N. Thus, we conclude that the more discriminative the principal directions are, the better the solutions. In extreme cases, when all embeddings in each class are identical to certain principal direction, the principal directions the optimal solution.
Evaluating k-means cluster centroids. Given the cluster membership indicators for k-means clustering defined in 13 i.e., He [hi....,he) E NxC, the cluster centroids are
h
he
V Z
(60)
Vni
VTC
Z ZHidi
{mn]
. n
(61)
where h2 consists of n. I's, with all other entries being zero. Then we have
Vdiag(nj
n T = ZH zw,
(62)
W HCT,
(63)
where T CCC is an orthogonal matrix whose last column is ( (VRI/N, .... Vne/N).. By
Theorem 3.1 of 13 we have
ddiag(nj.... T zw jzz 4 / 22, 9 9c-1/18-11
(64)
[X] .....,, 1 1--1c-110
(65)
20

---

Therefore, we have
[ dn
x ---1---11 OjT diag (nf
"ch:
(66)
which bridges the principal directions and the centroids of the & mean clusters. Then for the same reason of (55 and 57 we have
Ac
(67)
We conclude that the better the image embeddings (C' - 1) dimmennsion subspace, the more optimal the k-means cluster centroids become. In the extreme case where the image embeddings lie perfectly within a (C' -imimennio subspace, the k-means cluster centroids are the optimal solution.
D.3 Understanding self-attention applied to the concatenation of Z and Q
Given image embeddings Z DxD and class embeddings Q E DxC we concatenate them as Z Zz Q] and then, without loss generality, conduct non-parametric self-attention it, i.c.,
Z softmax Z Z = |Z. softmax (
2 2 2 o] Q ②
(68)
To simplify the analysis, we remove the softmax operator. As softmax computes along the last dimension (i.e., each column) by default, it essentially scales the update of each embedding. And since we interpret attention as a gradient step, this scaling does not alter the corresponding objective. We thus have
z z' zet o²
Z = -al'' Q4
(69)
Q Q' z' q Q'
1z++1 Q q iz', Q- al z' z + QQq'" z' z' Q q'Q Q'
(70)
Iz' a (z' z' z' + Q' z)) Q' a'z'z' Q q' Q*))
(71)
In Eq. there are four gradient terms: the terms z'z z' and Q'Q² Q' represent self-
attention, and the terms z' z' Q represent cross-attention. These have all been discussed in Section
3.3 and Section 3.4 However, the objective corresponding to Q'Q²" z remains unknown us.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The justification for contribution 1 Section 3.2 contribution 2 in Section and contribution 3. in Further details are provided in the Appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations, which also inform future work, in Section 5 and in Appendix [ we demonstrate the theoretically limitations through derivations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer: [Yes]
Justification: The main line of our derivations is related Section B as well as their assumptions. We provide detailed proofs in Appendix [ And most of our assumptions and conclusions are proved by the experiments we conduct.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our designs are clearly formulated in Section 3 and shown in Figure More
experimental details are in Appendix and we refer readers [35] and 42 for more background knowledge.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
23

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [No]
Justification: We release our code and model at httts///ggthhu...omm///aaaaa/a/a mentioned in abstract.
as
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https::/nnis.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wwtpcc/pccppcpcceeeooooooop gwwcc//guues///eee hutec/ggguudeee/eeeuuuubiiiiiiii wwwtinsssonnppo ssinnoliccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details our settings is in Appendix B and we refer readers to or our code for more relevant default settings.
35
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not reported because would be too computationally expensive for us.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: The effectiveness of DEPICT is robust with varying computer resources and we have controlled variables for fair comparisons. Therefore, we choose not to report the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httts//ttt/r///////p/oo////////t
Answer: [Yes]
Justification: Our research in the paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work focuses on deriving a white-box decoder for semantic segmentation, the work has societal impact.
Guidelines:
25

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our proposed white-box decoder for semantic segmentation poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Only publicly available assets are used in this paper, and we have referenced their owners.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
26

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work focuses on deriving a white-box decoder for semantic segmentation, there is human subjects involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work focuses on deriving white-box decoder for semantic segmentation, there is no human subjects are invovled.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "EFqWWBl1XB", "reviewer": "Reviewer_CNaF", "summary": "A view of compression from feature space with dimension $m$ to category $c$ is proposed for semantic segmentation. With this formulation of a PCA-like operation, a white-box encoder is designed with self-attention or cross-attention. Experiments show that the proposed encoder with ViT gets better performance than Maskformer.", "questions": "1. Is it possible to adapt the PCA analysis to the CNN-based model?\n2. The number of hyperparameters of the transformer block is very few. If the dim_head and heads are set to 1 and $c$ in the white-box encoder, will it limit the diversity of the model?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The view of compression for semantic segmentation is interesting and it is verified by visualization of the classifier $P$.\n2. Experiments verify the effectiveness of the encoder.", "weaknesses": "1. Compression is not the original idea of this paper and it is inspired by [37], so I do not give a very high review score. However, I still think this work is good.\n2. It's better to give results on another dataset to show the universal ability.\n3. It lacks comparison with the new SOTA semantic segmentation approaches. Only the maskformer is compared.\n4. \"A s\" in Line 241 --> \"As\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of our work. As suggested, we add experiments on more datasets, including Cityscapes and Pascal Context, and find that our DEPICT performs consistently better that of its black-box counterparts, i.e., the decoder of Segmenter. And we would like to compare our DEPICT to more advanced methods, showing that our DEPICT serves as an extremely light-weight decoder with acceptable lower mIoU. Specifically, we express our apologies for a typo that mis-spelled \"Mask Transformer\" as \"MaskFormer\" in the paper. Since that \"Mask Transformer\" has been widely used in various papers, we would use \"Segmenter's decoder\" instead in the final version of our paper.\n\nAs reported in details in our Common Responses, we have provided evaluations on more datasets to improve the empirical evaluations as Reviewer CNaF suggested. To make a fair comparison, we strictly implemented the *MSSA* block (see Line 171 for previous implementation) and removed the *ISTA* block (see Line 197 for context). These experimental results show that our DEPICT now consistently outperforms Segmenter's decoder with further reduced amount of parameters. While the white-box nature of our DEPICT has been strengthened by the above-mentioned modifications, we further conduct more experiments to validate our interpretation.\n\nHere, we focus on reporting how we have addressed the concerns listed in the review comments.\n\nAs raised by reviewer, the idea of quantifying compression by coding rate is indeed not the novelty of our work and, we did not claim that as one of our novel contributions. To make our novel contributions more clear, we would like to make the following interpretation (or clarification). Compared to [1], our novelty and distinct contributions are:\n+ We introduce the idea of compression to design an interpretable semantic segmentation and propose an interpretation framework from the perspective of the classical *PCA*.\n+ For the first time, we derive the cross-attention variant of the MSSA block and we conduct experiments to evaluate the performance of its strict implementation, showing improved performance (also see B.1.1 of [1] for its relaxed implementation). \n\nSecond, we would like to report more experimental results on datasets Cityscapes and Pascal Context.  Due to limited time during rebuttal phase, we do not report the results of ViT-Large based DEPICT on these two datasets at this moment, but we will get it done soon (during the discussion, or at least in the final version). We report the results based on ViT-Small, and show that our DEPICT again outperforms Segmenter by 7% mIoU on both Cityscapes and Pascal Context.\n\nThird, as suggested by reviewers, we would like to provide a set of direct comparisons between DEPICT and more advanced black-box methods. Nevertheless, we have to argue that such kind of comparisons currently are unfair for our DEPICT. For example, on ADE20K, MaskFormer's Swin-L based decoder achieves 54.1\\% mIoU with at least 270 GFLOPs; while our ViT-L (lagging behind Swin-L by 2% top1 acc) based DEPICT achieves 52.5\\% mIoU with merely 10 GFLOPs. Meanwhile, MaskFormer uses a linear combination of the focal loss and the dice loss; whereas our DEPICT merely uses the standard CE loss without weight rebalancing. If there is no time limitation, we would like to attempt to address all these issues for fair comparisons. \n\n- For Question #1: \"Is it possible to adapt PCA to CNN-based models?\" This is an insightful but challenging question. It seems to us that the answer is yes. According to [2], a self-attention layer can express any convolutional layer. Conversely, a self-attention operation may be decomposed into convolutional filters. Additionally, in fully convolutional networks, classification presents as features output by the last layer. In other words, there seems no difference between extracting features and giving classification, for that is right the reason allowing us to introduce PCA for interpretation.\n\n- For Question #2: we actually find that adding constraint $\\text{heads}*\\text{dim}$_$\\text{head}=C$ is unreasonable, as its derivation considered only one image whereas there are various principal directions across images of the whole dataset. Therefore, we removed this constraint. Then, we conduct experiments to evaluate the impact of using a ranging number of heads, and find that a relatively small number of heads (i.e., 3 for ADE20K) renders the best performance, implying that a tighter lower bound leads to less effective training. Using a large number of heads not only causes a heavier occupation on GPU memory but also results in inferior performance. Please let us know if we misunderstand your question. Thanks so much!\n\nWe hope that our point-by-point responses above have resolved your concerns on our work and, highly appreciate your encouragement in the comments and the pre-rating. Please let us know if any further clarification is needed.\n\n[1] White-box transformers via sparse rate reduction. Yaodong Yu, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Benjamin Haeffele, Yi Ma. NeurIP 2023.\n\n[2] On the relationship between self-attention and convolutional layers. Jean-Baptiste Cordonnier, Andreas Loukas, Martin Jaggi. ICLR 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hxmIxEJKeT", "reviewer": "Reviewer_ZoMh", "summary": "This work attempts to view Transformer-based semantic segmentation from the perspective of principal component analysis and develops interpretable Transformer decoders for segmentation based on this insight, which can achieve comparable performance to their black-box counterparts with significantly fewer parameters.", "questions": "See weakness.", "limitations": "Although the paper presents a very fascinating theoretical framework, the authors' ignorance and misunderstanding of the basic Transformer-based segmentation models make many claims in this paper is incorrect. And the formulation of the baseline black-box this manuscript focus on is not sufficiently representative.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The presentation is overall clear and the derivation is solid;\n2. It is impressive to achieve comparable performance with fewer parameters and interpretability;\n3. The visualization of learned parameters in derived architecture validates the assumption and derivation.", "weaknesses": "1. The authors seem to lack awareness of recent advances in segmentation and have significant misunderstanding about Transformer-based decoders for segmentation. L34-35 says \"these decoders typically set a learnable class embedding for each predefined class.\" However, in the four citations at L78 ([27,6,5,13]), only the first paper adopts this paradigm. The other works do not include learnable class embeddings for predefined classes but use “zero-initialized query features with positional embeddings corresponding to instances/segments, instead of classes. Thus, the formulation in this paper does not apply to any of them. Additionally, it is incorrectly stated in L246 that MaskFormer serves as the decoder for Segmenter.\n2. The comparison in Figure 6 does not use the most advanced methods as the baseline, making the results less convincing. Besides, the linear decoder achieves performance comparable to the black-box counterpart with only about one percent of the parameters. This somewhat implies that the results presented in this work are trivial.\n3. The authors claim that there is principled guidance to set hyper-parameters such as num_heads and dim_heads. However, does this constraint ensure consistently satisfactory performance without the need for laborious manual tuning? Or is it merely to reduce the hyper-parameter tuning space to align with theoretical derivations? If it is the former, it would be beneficial to report results on a broader range of datasets to demonstrate its generality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. It is indeed that the meta-architecture we investigated sounds not representative enough for a vast range of Transformer-based decoders. However, the insights of our work is capable to be generalized to partly interpret them. \n\nFirst of all, we express our sincere appreciation for pointing out our mistakes. We will adopt the meta architecture proposed by [1] (see its Fig.3) as the truly representative one and admit that our interpretation is investigating a simplified one in the paper. Additionally, we will replace \"MaskFormer\" with \"Segmenter\" in the table. The decoder of Segmenter is called \"Mask Transformer\" but we mistakenly spelled it. Considering that, \"Mask Transformer\" has been widely used in various paper, we would not to use it in the final version of the paper in order to avoid naming ambiguity.\n\nNow, we focus on reporting how we attempt to address the concerns listed by reviewer.\n\n- We would like to show that addressing a limitation of our current interpretation will result in replacing class embeddings with segment/instance embeddings. That is, using class embeddings couples each class with a principal direction, such that each class strictly corresponds to one principal direction. However, an image typically contains a small part of all classes and each class allows rich intra-class variance thus demands more than one principal directions to representing itself. Therefore, it is more reasonable to introduce segment or instance as a finer-grained and more flexible concept to replace the current role of class.  As evidence, we find that there are still higher correlations among class embeddings output by our DEPICT. Although  such correlations can be significantly eased by using cross-attention, it leads to a drop in mIoU. Additionally,  Mask2Former reports that making query features learnable raises mIoU by 1.8% (see its Table 4. (b)). From the perspective of our work, it is because that the attention block performs a gradient descent step and a good initialization to start would be beneficial. \n\n- As suggested by reviewers, we would like to provide a set of direct comparisons between DEPICT and more advanced black-box methods. Nevertheless, we have to argue that such kind of comparisons currently are unfair for our DEPICT. For example, on ADE20K, MaskFormer's Swin-L based decoder achieves 54.1% mIoU with at least 270 GFLOPs; whereas our ViT-L (lagging behind Swin-L by 2% top1 acc) based DEPICT achieves 52.5% mIoU with merely 10 GFLOPs. Meanwhile, MaskFormer uses a linear combination of the focal loss and the dice loss; whereas our DEPICT merely uses the standard CE loss without weight rebalancing. If there is no time limitation, we would like to attempt to address all these issues for fair comparisons. As for the linear decoder, it is not scalable not only on layer backbone but also to trade the amount of parameters for better mIoU. For example, it lags behind our DEPICT by 1.8% mIoU when based on ViT-Large and simply adding more linear layers does not change the performance. (These results are based on our newly modified DEPICT, and have reported in our Common Response.)\n\n- We actually find that adding constraint $\\text{heads}*\\text{dim}$_$\\text{head}=C$ is unreasonable, as its derivation considered only one image whereas there are various principal directions across images of the whole dataset. Therefore, we have removed this constraint. Then, we conduct experiments to evaluate the impact of using a ranging number of heads, and find that using a relatively small number of heads (i.e., it is 3 for ADE20K) leads to the best performance, implying that a tighter lower bound leads to less effective training. \n\n- We report more experimental results on Cityscapes and Pascal Context. Limited by time, We did not report the result of ViT-Large based DEPICT on these two datasets, but we will get it done soon (during the discussion phase or at least in the final version). Our DEPICT based on ViT-Small can outperform Segmenter by 7% mIoU on both Cityscapes and Pascal Context.\n\nWe hope that our clarification points above could resolve your concerns on our work. Please let us know if any further clarification is needed. \n\n[1] Transformer-Based Visual Segmentation: A Survey. Xiangtai Li, Henghui Ding, HaoboYuan, Wenwei Zhang, Jiangmiao Pang,  Guangliang Cheng, Kai Chen, Ziwei Liu, Chen Change Loy. TPAMI 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eMh9BF9PIk", "reviewer": "Reviewer_MHpG", "summary": "This paper derives white-box decoders for Transformer-based semantic segmentation from a compression perspective, which links the process to Principal Component Analysis (PCA). The authors introduce DEPICT, which clarifies the mechanisms of black-box decoders and achieves comparable performance with significantly fewer parameters. Using ViT-B as the encoder, DEPICT outperforms traditional mask transformers on the ADE20K dataset.", "questions": "1.\tPlease include the Cityscapes dataset, which is widely used in semantic segmentation, in your experiments.\n2.\tCould you compare the performance of the proposed method with the following prominent Transformer decoder-based methods to see if your approach can be applied and how it performs against them?\n\t1)\tSegFormer (Xie et al. 2021)\n\t2)\tSegmenter (Strudel et al. 2021)\n\t3)\tMask2Former (Cheng et al. 2022)\n\t4)\tFeedFormer (Shim et al. 2023)\n3.\tIn Transformer decoder structures, GFLOPs (floating-point operations per second) are more crucial than the number of parameters due to the attention mechanism. Please include GFLOPs in your comparisons.\n\nIf these points are addressed during the revision period, I would be willing to consider increasing my evaluation score.", "limitations": "The most significant limitation of this paper is the lack of extensive experiments to validate the performance of the proposed algorithm. Additionally, the paper does not consider the FLOPs metric, which is crucial for evaluating efficiency. \nPlease address the points mentioned in the questions section to strengthen the validation of your proposed DEPICT algorithm.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper provides a solid theoretical foundation by analyzing mathematically complex decoders used in Transformer-based semantic segmentation. \n\nThis method bridges the gap between theoretical understanding and practical application. \n\nIn addition, DEPICT decoders achieve comparable performance to traditional black-box decoders while using significantly fewer parameters.", "weaknesses": "The primary drawback of this paper is the lack of extensive experimentation with existing decoder architectures. While the focus on theoretical derivation is understandable, the current experimental results are insufficient to confirm that the proposed DEPICT algorithm operates as claimed by the authors. A more comprehensive comparison across various datasets and architectures is necessary to validate the effectiveness and generalizability of DEPICT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions. As the reviewer suggested, we have further validated our proposed interpretation and evaluated the effectiveness of our DEPICT on a set of new experiments on datasets Cityscapes and Pascal Context, including ablating the variants, testing the robustness under parameter perturbation and evaluating GFLOPs. We report the performance of our modifications to DEPICT, their justifications, detailed experimental results and an outline of our interpretation in our Common Response. \n\nHere, we address the concerns listed by reviewer MHpG.\n\n- First of all, we evaluate the GFLOPs, the amount of floating point operations, of only the decoders for fair comparison. Compared with Segmenter's decoder (Mask Transformer), the GFLOPs of our DEPICT is 2/3/4/10 versus Mask Transformer's 2/6/22/37 when both based on ViT-Tiny/Small/Base/Large. Compared to MaskFormer's decoder, the GFLOPs of our DEPICT is 10 based on ViT-Large; whereas it is at least 270 for MaskFormer based on Swin-Large (total: 375 GFLOPs [1] - Swin-L: more than 104 GFLOPs [2]). These evidences demonstrate that our DEPICT indeed serves an extremely light-weight decoder with comparable even better mIoU. Specifically, we must express our apologies for a vital typo that we mis-spelled \"Mask Transformer\" as \"MaskFormer\" in our submitted paper. Considering that, \"Mask Transformer\" has been widely used in various papers, we would use \"Segmenter's decoder\" instead in the final version of our paper. \n\n- Then, we report our newly added experimental results on Cityscapes and Pascal Context.  Due to limited time during rebuttal phase, we did not report the results of ViT-Large based DEPICT on these two datasets, but we will get it done soon (during the discussion period or at least in the final version). Based on ViT-Small, we can observe that our DEPICT outperforms Segmenter by 7% mIoU on both Cityscapes and Pascal Context.\n\n- Third, as suggested by reviewers, we would like to provide a set of direct comparisons between DEPICT and more advanced black-box methods. Nevertheless, we  argue that such kind of comparisons currently are unfair for our DEPICT. For example, on ADE20K, MaskFormer's Swin-L based decoder achieves 54.1% mIoU with at least 270 GFLOPs; whereas our ViT-L (lagging behind Swin-L by 2% top1 acc) based DEPICT achieves 52.5% mIoU with merely 10 GFLOPs. Meanwhile, MaskFormer uses a linear combination of the focal loss and the dice loss; whereas our DEPICT merely uses the standard CE loss without weight rebalancing. If there is no time limitation, we would like to attempt to address all these issues for fair comparisons. In addition, \nwe now report better performance of our newly modified DEPICT, compared to Segmenter (the only fair comparison). Details can be found in our Common Response.\n\n- For applying our interpretation to more black-box methods, it seems a potential but challenging future work. The most thorny part is prevalent methods typically involving a pixel decoder (see Fig.3 of [3]), resulting in extremely more GFLOPs and parameters, and remaining uninterpretable from our proposed perspective. Our work implies that there must exist more efficient methods without the burdensome pixel decoder. \n\nMost importantly, as suggested by the reviewer, we add several interesting experiments to confirm that our DEPICT indeed performs as we claimed. Here, we would like to list all current evidences.\n\n+ We strictly implement the *MSSA* block (see Line 171 for previous implementation) and remove the *ISTA* block (see Line 197 for context), enabling us to conclude that all DEPICT does is maximizing the projected coding rate on subspaces spanned by leading principal directions or minimizing it on subspaces spanned by remaining principal directions. By the sign of the learned step-size parameter, it is a white box for us to find out whether a specific block is maximizing it or minimizing it.  \n\n+ According to above point, all the learned step size should be positive if we let all blocks model the ambient space rather than low-dimensional subspaces. We conduct a set of experiments in this extreme case by setting $\\text{heads}=1$ and $\\text{dim}$_$\\text{head}=\\text{dim}$, and the result is as expected.\n+ We visualize all the class embeddings output by a cross-attention block across images and find that principal directions lies in a union of $C$ orthogonal subspaces.\n+ During the inference phase, we generate the $\\text{heads}$ random orthogonal matrices per image and use them to transform the parameter matrices of each head individually. We observe that DEPICT based on various ViT backbones shows no accuracy drop under such a perturbation. Furthermore, we generate one random orthogonal matrix to transform the parameter matrix of the entire block. Despite parameters are perturbed across heads, DEPICT based on various ViT backbones still shows limited drop in mIoU (less than 3% for ViT-L based DEPICT). Such robustness strongly validate the interpretation that our attention blocks essentially model subspaces thus performing orthogonal transformations on their learned bases will not lead to collapsed performance.\n\nWe hope  the clarifications above could resolve your concerns on our work. Please let us know if any further clarification is needed. \n\n\n[1] Per-Pixel Classification is Not All You Need for Semantic Segmentation. Bowen Cheng, Alex Schwing, Alexander Kirillov. NeurIPS 2021.\n\n[2] Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. Ze Liu, Yutong Lin, Yue Cao,  Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo. ICCV 2021.\n\n[3] Transformer-Based Visual Segmentation: A Survey. Xiangtai Li, Henghui Ding, HaoboYuan, Wenwei Zhang, Jiangmiao Pang,  Guangliang Cheng, Kai Chen, Ziwei Liu, Chen Change Loy. TPAMI 2024.\n\n[4] White-box transformers via sparse rate reduction. Yaodong Yu, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Benjamin Haeffele, Yi Ma. NeurIP 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MK8RYQIeWO", "reviewer": "Reviewer_V3V2", "summary": "This paper introduces a novel perspective by conceptualizing semantic segmentation as data compression, akin to PCA, which simplifies the role of decoders in Transformer-based models. It presents DEPICT, white-box decoders that clarify the functions of learnable class embeddings, self-attention and dot-product operations, achieving comparable or better performance with fewer parameters on the ADE20K dataset compared to traditional methods. This work enhances the understanding of decoder mechanisms in semantic segmentation.", "questions": "1. The main concern is the brevity of the experimental section, which focuses solely on the ADE20K dataset. It is necessary to expand experimental validation to include a variety of datasets and different model architectures to thoroughly evaluate the generalizability and robustness of the proposed methods.\n\n2. The proofs within the paper are notably difficult to comprehend, and due to specific issues in detail and explanation as mentioned, their rigor cannot be assured. This ambiguity affects the verification of their correctness. It is crucial for the authors to not only address the aforementioned concerns but also to refine the presentation of these proofs to enable readers to effectively assess their accuracy.", "limitations": "The paper acknowledges its limitations in the appendix. There is no discussion of potential negative societal impacts, as they are not applicable to this technical research.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The figures and tables in the paper are informative.\n2. The experiments validate that the proposed methods reduce model parameters while achieving adequate performance.\n3. The paper provides thorough mathematical derivations of the related theories.", "weaknesses": "1. The experimental validation is limited as it only involves a single dataset, ADE20K. Expanding the testing to include a variety of datasets along with a detailed analysis of FLOPs would provide a stronger evidence of the methods' efficiency and adaptability.\n\n2. Some mathematical proofs are confusing or inadequately explained:\n   - **Line 144**: The assertion of strict equivalence lacks a thorough explanation, although it may seem intuitively correct.\n   - **Equation (4)**: The text does not clearly explain the implications of transposing elements $(p_c^T Z)^\\top$ and $(p_c^T Z)$, especially in terms of how the unit matrix is affected, which could lead to dimensional inconsistencies.\n   - **Line 412**: The equivalence between minimizing $|log(1 + \\beta)|$ where $\\beta > 0$ and minimizing $|\\beta – 1|$ is confusing, as logically the former minimizes when $\\beta$ approaches zero, whereas the latter minimizes when $\\beta$ equals one.\n\n3. Figure 2's subfigures (b) and (c) appear identical apart from font color, and the caption provides no explanation, raising questions about whether this is an oversight or intended to convey a specific message.\n\n4. The claim on Line 50 that semantic segmentation can be viewed as PCA is too broad and not specific to the novel contributions of this paper, as it could apply broadly to any task involving classification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate your valuable comments, especially for your attention to the details in our paper.\n\n**Response to Weakness #1.** We conduct more experiment and provide the new experimental results on Cityscapes and Pascal Context (in Common Response).  Due to time limitation, we did not provide ViT-Large based DEPICT on these two datasets (at this moment), but we will get it done soon (maybe during the discussion or at least in the final version). Based on ViT-Small, we show that our DEPICT outperforms Segmenter by 7% mIoU on both Cityscapes and Pascal Context. Regarding the GFLOPs metric, our decoder requires remarkably fewer GFLOPs as it is fully attention based method and utilizes the MSSA block. The GFLOPs of our decoder is 2/3/4/10 based on ViT-Tiny/Small/Base/Large, whereas the segmenter's is 2/6/22/37. \n\n**Response to Weakness #2.** The conclusion of line 144 is justified by that maximizing a scalar $x$ is equivalent to maximizing $\\log(1+\\alpha x)$ where $0 \\leq x \\leq 1$, $\\alpha > 0$. The identity matrix of equation (3) degenerate to scalar 1, and the identity  matrix of equation (4) has a shape of $N \\times N$, where $N$ is the sequence length. Despite the dimensions are inconsistent, equation (3) is strictly equivalent to equation (4), by Lemma A.2 of [1]. For line 412, it is indeed a mistake, which should be corrected as \"being equivalent to minimizing $|\\beta|$\". Thanks for pointing it! Fortunately, this mistake doesn't affect the correctness of the following equation (37). We are willing to checking or refining our proofs. For example, we have proposed much more concise and intuitive proofs from the perspective of low-rank approximation (see Response to Question#2). \n\n**Response to Weakness #3.** It is our fault for providing no explanation about the font color. The red color means that the parameters are somewhat constrained as we constrained that $\\text{heads}*\\text{dim}$_$\\text{head}=C$. However, as explained in our Common Response, we now remove this constraint, as it only holds for a single image rather than for the whole dataset. \n\n**Response to Weakness #4.** This is a very insightful concern. A model for semantic segmentation processes all the patch embeddings of a singe image and tries to classify each of them, i.e., both the transformation and the classification are done at the patch-level. Such consistency enables us to interpret it by the idea of PCA. In other words, as models are typically only allowed to make direct information exchange within pixels or patches of a singe image, image segmentation is the most suitable task for applying PCA. Although based on semantic segmentation, our proposed interpretation framework can be applied to other dense prediction tasks. For example, we can allow that a class can be represented by more than one principal directions and each of these principal directions stand for an object or a segment, resulting in object detection and instance segmentation. However, it is challenging to adopt our interpretation from pixel-level tasks to image-level tasks. But, we believe that compression is an inevitable perspective for many tasks and our work serves as a first yet a solid step toward developing a comprehensive interpretation framework. \n\n**Response to Question #1.** Please refer to Response to Weakness #1. \n\n**Response to Question #2.** Thank you for point out that our previous proofs are difficult to comprehend. To address this issue, we now adopt a classical perspective of low-rank approximation to prove that class embeddings are transformed to be the principal directions by the attention operations. Please refer to the Interpretation Outline section of our Common Response for our rephrased thoughts and conclusions. In short, we prove that the principal directions are a good low-rank approximation to the patch embeddings in terms of coding rate. Therefore, we replace equation (20) with $\\min_{\\boldsymbol{Q}} |R(\\boldsymbol{Z})-R(\\boldsymbol{Q})|$ $\\text{s.t}.$ $\\text{rank}(\\boldsymbol{Q})=C$, which is a more intuitive objective motivated by the idea of low-rank approximation. By the geometric interpretation of the coding rate, we will have that $R(\\boldsymbol{Z}) \\geq R(\\boldsymbol{Q})$, constraining that each class embeddings lie in the subspace of patch embeddings. Therefore, the objective becomes $\\max_{\\boldsymbol{Q}} R(\\boldsymbol{Q})$, and it derives a self-attention-like operation on $\\boldsymbol{Q}$. For keeping the constraint and building up the connection between $\\boldsymbol{Q}$ and $\\boldsymbol{Z}$, we can replace the first and the second $\\boldsymbol{Q}$ in the self-attention $\\boldsymbol{Q}(\\boldsymbol{Q}^T \\boldsymbol{Q})$ with $\\boldsymbol{Z}$ and get $\\boldsymbol{Z}(\\boldsymbol{Z}^T \\boldsymbol{Q})$, a cross-attention-like operation. Notice that the output of such an operation always lies in the subspace of $\\boldsymbol{Z}$. It is easy to prove that the scaled leading $C$ principal directions are an optimal solution for the above low-rank approximation objective. Although we believe that current line of thought is intuitive, we are willing to provide detailed proofs if reviewers request.\n\n[1] Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction. Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, Yi Ma. NeurIPS 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for the constructive suggestions and insightful comments. We appreciate the weaknesses and limitations spotted by reviewers. To address these issues, we would like to provide more theoretical and experimental results. Briefly, we further solidify our compression-based interpretation by presenting more rigorous proofs, and achieve much better experimental results while strictly implementing the derivations. As requested by reviewers, we conduct new experiments to further validate the effectiveness and mechanism of our proposed white-box decoders.  

+ For compression with less information loss, we maximize *the projected coding rate* respectively on $C$ principal directions. As it is costly to directly maximize them, we assumed that the projected coding rate on a subspace spanned by them serves as the lower bound of their sum. Now, we rigorously prove this point holds. 

+ Aligning with [1], the utilized *MSSA* block involved an independently learned parameter matrix for implementation simplicity (see B.1.1 in [1]). We now remove it to guarantee the model functions as our interpretation suggests. Meanwhile, as the *ISTA* block lacked convincing justification for its utilization, it is also removed, meaning that DEPICT is now fully attention-based. While strengthening the interpretability of our proposed DEPICT, these two modifications also significantly improve the segmentation performance and reduce the parameter amount. 

+ By the decoder, class embeddings representing the same class vary across images. Such variability is desirable for allowing rich intra-class variation and partly explain why query decoder is superior to linear decoder. Back to our work, we constrained $\text{heads}* \text{dim}$\_$\text{head}=C$ as there are at most $C$ classes and each of them is represented by one principal direction. However, it merely holds for a single image. For the whole dataset, class embeddings lie in a union of $C$ orthogonal subspaces and each subspace is spanned by the class embeddings representing a specific class. Therefore, we remove the constraint that $\text{heads}* \text{dim}$\_$\text{head}=C$. 

+ We now integrate the proof for solving principal directions into the coding rate framework by providing a more generalized and insightful proof from the perspective of *low-rank approximation*. In short, the scaled principal directions approximate patch embeddings well in terms of coding rate. 

# Key Contributions
Our central contribution is that we introduce a compression-based interpretation framework for Transformer-based semantic segmentation decoders. Although the proofs are complicated, the conclusions are simple and intuitive: self-attention prepares patch embeddings for compression and cross-attention seeks principal directions as classifiers. Based on such insights, we design white-box decoders, DEPICT, which remarkably outperform their black-box counterparts in terms of mIoU, #params, and GFLOPS. In summary, our work serves as a first yet solid step toward developing a comprehensive interpretation framework for semantic segmentation. We believe that our insights and findings are worth sharing with the community to encourage further efforts toward better white-box decoders.

# New Experiments
Here, we provide a brief summary of the new experiments and their results for the reviewers. We compare our modified DEPICT with Segmenter on ADE20K, Cityscapes, and Pascal Context, and find that DEPICT consistently performs better. We test the robustness of DEPICT under parameter perturbation, and observe that DEPICT shows very limited drop in mIoU. We ablate variants of DEPICT and find that the currently adopted fully attention-based design performs the best. Please refer to the attached .pdf for detailed illustrations and results. We are willing to provide more detailed experimental results if requested by the reviewers.

[1] White-box transformers via sparse rate reduction. Yaodong Yu, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Benjamin Haeffele, Yi Ma. NeurIP 2023.

Meta-review: This paper derives white-box decoders for Transformer-based semantic segmentation from the perspective of compression.
Initially it received rating of 6544. 
Concerns are about theoretical aspects and more experiment validation.
Most of the concerns were resolved during rebuttal and the reviewers raised the score to 6555.
The AC recommends acceptance for this paper.
The authors are urged to include more comprehensive experiments (eg, Cityscapes) as requested by reviewer V3V2 and reviewer MHpG in the camera-ready version of the paper.
Decision: Accept (poster)