Multi-Scale Representations by Varying Window Attention for Semantic Segmentation
Authors: Haotian Yan
Ming Wu
Chuang Zhang
Venue: ICLR 2024 poster (2024)
Keywords: Semantic Segmentation
Multi-Scale Representations Learning
Attention
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Multi-scale learning is central to semantic segmentation. We visualize the effective receptive field (ERF) of canonical multi-scale representations and point out two risks learning them: \textit{scale inadequacy} and \textit{field inactivation}. A novel multi-scale learner, \textbf{varying window attention} (VWA), is presented to address these issues. VWA leverages the local window attention (LWA) and disentangles LWA into the query window and context window, allowing the context's scale to vary for the query to learn representations at multiple scales. 
However, varying the context to large-scale windows (enlarging ratio $R$) can significantly increase the memory footprint and computation cost ($R^2$ times larger than LWA). We propose a simple but professional re-scaling strategy to zero the extra induced cost without compromising performance. 
Consequently, VWA uses the same cost as LWA to overcome the receptive limitation of the local window.
Furthermore, depending on VWA and employing various MLPs, we introduce a multi-scale decoder (MSD), \textbf{VWFormer}, to improve multi-scale representations for semantic segmentation. VWFormer achieves efficiency competitive with the most compute-friendly MSDs, like FPN and MLP decoder, but performs much better than any MSDs. 
For instance, using nearly half of UPerNet's computation, VWFormer outperforms it by $1.0\%-2.5\%$ mIoU on ADE20K.
At little extra overhead, $\sim 10$G FLOPs, Mask2Former armed with VWFormer improves by $1.0\%-1.3\%$.

Published as a conference paper at ICLR 2024
MULTI-SCALE REPRESENTATIONS BY VARYING WINDOW ATTENTION FOR SEMANTIC SEGMENTATION
Haotian Yan; Ming Wu' Chuang Zhang
Artificial Intelligence School
Beijing University of Posts and Telecommunications, China (yanhaot an, wuming, zhangchuang) ebupt..du..uttu.u......
ABSTRACT
Multi-scale learning is central to semantic segmentation. We visualize the effec- tive receptive field (ERF) of canonical multi-scale representations and point out risks learning them: scale inadequacy and field inactivation. A novel multi- scale learner, varying window attention (VWA). is presented to address these issues. VWA leverages the local window attention (LWA) and disentangles LWA into the query window and context window, allowing the context's scale to vary multiple scales. However, varying the
for the query to learn representations context to large-scale windows (enlarging ratio R) can significantly increase the memory footprint and computation cost (/t² times larger than LWA). We propose simple but professional re-scaling strategy to zero the extra induced cost with- out compromising performance. Consequently, VWA uses the same cost as LWA overcome the receptive limitation of the local window. Furthermore, depend- ing on VWA and employing various MIPs, we introduce a multi-scale decoder (MSD), VWFormer, to improve multi-scale representations for semantic seg- mentation. VWFormer achieves efficiency competitive with the most compute- friendly MSDs, like FPN and MLP decoder, but performs much better than any MSDs. For instance, using nearly half of UPerNet's computation, VWFormer out- performs by 1.0% 2.5% mloU on ADE20K. At little extra overhead, 10G FLOPs, Fask2rormer armed with VWFormer improves by 1.0% 1.3%.
INTRODUCTION
In semantic segmentation, there are typical paradigms for learning multi-scale representa- tions. The first involves applying filters with receptive-field-variab kernels, classic techniques like atrous convolution (Chen hyper-parameters. such as dilation rates and pooling output sizes, the network can vary the receptive field to learn representations at multiple scales.
2018) or adaptive pooling (Zhao al. 2017). By adjusting
The second leverages hierarchical backbones Xie et al. (2021) > Liu et al. 2021 2022) to learn multi- scale representations. Typical hierarchical backbones are usually divided four different levels, each learning representations on feature maps with different sizes. For semantic segmentation, the
multi-scale decoder (MSD) (Xiao al. 2018 Kirillov et al. 2019 Xie et al.. (02T) fuses feature maps from every level (i.e. multiple scales) and output an aggregation of multi-scale representations.
Essentially, the second paradigm is analogous to the first in that it can be understood from the per- spective of varying receptive fields of filters. As the network deepens and feature map sizes gradually shrink, different stages of the hierarchical backbone have distinct receptive fields. Therefore, when MSDs work for semantic segmentation, they naturally aggregate representations learnt by filters with multiple receptive fields, which characterizes multi-level outputs of the hierarchical backbone.
* "The code and model will be available at 1 Contributed Equally to the First Author. 1 'Coresponding Author.

---

Published as a conference paper at ICLR 2024
(a) ASPP: atrous rales from left to right (1, 12, 24, 36)
(b) PSP: oulput size from left to right (1. 2,
ConvNeXt: stagel, stage2, stage3, stage4
(d) Swin Trunsformer. stagel, stage2, stage3, stage4
(e) SegFormer: stagel, stage2, stage3, stage4
Our proposed VWA: residual path, R-4,
Figure ERFs of multi-scale representations learned by (a) ASPP, (b) PSP, (c) ConvNeXt, (d) Swin Transformer, (e) SegFormer, and (f) Our proposed varying window attention. ERF maps are visualized across 100 images of ADE20K validation set. See Appendix for more detailed analysis.
To delve into the receptive field of these paradigms, their effective receptive fields (ERF) Luo al, 2016) were visualized. as shown Fig. T.e For first paradigm, methods like ASPP (applying atrous convolution) Chen et al. (2018) and PSP Zhao et al. (2017) (applying adaptive pooling) were analyzed. For the second paradigm, ERF visualization was performed on multi-level feature maps of ConvNeXt (Liu et al. 2022). Swin Transformer Liu et al. (002), and SegFormer (MiT) Xie al. (2021). Based on these visualizations, it can be observed that learning multi-scale representations faces issues. On the hand, there is risk of scale inadequacy, such missing global information (Swin Transformer, ConvNeXt, ASPP), missing local information (PSP), or having only local and global information while missing other scales (SegFormer). On the other hand, there are inactivated areas within the spatial range of the receptive field, as observed in ASPP, Swin Transformer, and the low-level layers of SegFormer. We refer to this as field inactivation.
To address these issues, new way is explored to learn multi-scale representations. This research focuses on exploring whether the local window attention (LWA) mechanism can be extended to func- tion relational filter whose receptive field is variable to meet the scale specification for learning multi-scale representations semantic segmentation while preserving
efficiency advantages of LWA. The resulting approach is varying window attention (VWA), which learns multi-scale repre- sentations with no room for scale inadequacy and field inactivation (See Fig. DD. Specifically, VWA disentangles LWA into the query window and context window. The query remains positioned on the local window, while the context enlarged to cover more surrounding areas, thereby varying the receptive field of the query. Since this enlargement results in substantial overhead impairing the high efficiency of LWA (R²² times than LWA), we analyze how the extra cost arises and particularly devise pre-scaling principle, densely overlapping patch embedding (DOPE), and copy-shift padding mode (CSP) to eliminate it without compromising performance.
More prominently, tailored semantic segmentation, we propose multi-scale decoder (MSD), VWFormer, employing VWA and incorporating MLPs with functionalities including multi-layer aggregation and low-level enhancement. paired with versatile backbones such as ConvNeXt, Swin Transformer, SegFormer, and compare it with classical MSDs like FPN (Lin et al.) 2017). UperNet (Xiao et al.) 2018). MLP-decoder (Xie) et al. 2021), and deform-attention (Zhu et al., 2020) on datasets including ADE20K (Zhou et al.) 2017). Cityscapes (Cordts et al., 2016), and COCOStuff- 164k (Caesar et al. 2018). Experiments show that VWFormer consistently leads to performance and efficiency gains. The highest improve- ments can reach an increase 2.1% mloU and FLOPs reduction of 45% which are credited to VWA rectifying multi-scale representations of multi-level feature maps at costs of LWA.
prove the superiority of VWFormer, we evaluate
In summary, this work has a three-fold contribution:
We make full of the ERF technique to visualize the scale of representations learned by exist- ing multi-scale learning paradigms, including receptive-field variable kernels and different levels of hierarchical backbones, revealing the issues of scale inadequacy and field inactivation.
2

---

Published as a conference paper at ICLR 2024
We propose VWA, relational representation learner, allowing for varying context window sizes toward multiple receptive fields like variable kernels. is as efficient as LWA due to our pre-scaling principle along with DOPE. We also propose CSP padding mode specifically for perfecting VWA.
A novel MSD, VWFormer, designed for semantic segmentation, is presented as the product of VWA. VWFormer shows its effectiveness in improving multi-scale representations of hierarchical backbones, by surpassing existing MSDs in performance and efficiency on classic datasets.
RELATED WORKS
2.1 MULTI-SCALE LEARNER
The multi-scale learner is deemed the paradigm utilizing variable filters to learn multi-scale repre- sentations. Sec. hass introduced ASPP and PSP. There are also more multi-scale learners proposed previously for semantic segmentation. These works can be categorized into three groups The first involves using atrous convs, e.g. ASPP, and improving its feature fusion way and efficiency of atrous convolution (Yang et al.) 2018 Chen et al. 2018). The second involves extending adaptive pooling, incorporating PSP into other types of representation learners (He al.) 2019a) (He et al. 2019b). However, there are issues of scale inadequacy and field inactivation associated with these methods' core mechanisms, i.e. atrous convs and adaptive pooling, as analyzed in Scc.
The third uses similar idea ours, computing the attention matrices between the query and con- texts with different scales, to learn multi-scale representations a relational way for semantic seg- mentation even image recognition. In the case of Yuan et al. (2018) and Yu al. (202T) ) their core mechanisms are almost identical. As for Zhu et al. (2019). Yang et al. (2021). ) and Ren et al. (2022), the differences among the three are also trivial. We briefly introduce Yuan al. (2018) and Zhu et al. 2019)), visualizing their ERFs and analyzing their issues (See Fig. 7and Appendix B for more information). In word, all of the existing multi-scale learners in a relational way (also known as multi-scale attention) do not address the issues we find, i.e. scale inadequacy and field inactivation.
2.2 MULTI-SCALE DECODER
The multi-scale decoder (MSD) fuses multi-scale representations (multi-level feature maps) learned by hierarchical backbones. One of the most representative MSDs is the Feature Pyramid Network (FPN) (Lin et al., 2017). originally designed for object detection. It has also been applied image segmentation by using its lowest-level output, even in SOTA semantic segmentation methods such as MaskFormer (Cheng et al. al. 2019) and (Huang et al.
2021). [Lin et al. (2017) has also given rise methods like (Kirillov)
(Cheng et al. 2022). FPN is combined with deformable attention Zhu et al. (2020) to allow relational interaction between different level feature maps, achieving higher results. Apart from FPN and its derivatives, other widely used methods include the UperNet (Xiao et al.
2021). In Mask2
2018) and the lightweight MLP-decoder proposed by SegFormer.
In summary, all of these methods focus on how to fuse multi-scale representations from hierarchical backbones or enable them to interact with each other. However, our analysis points out that there are scale inadequacy and field inactivation issues with referring to multi-level feature maps of hierarchi- cal backbones as multi-scale representations. VWFormer further learns multi-scale representations with distinct scale variations and regular ERFs, surpassing existing MSDs in terms of performance while consuming the same computational budget as lightweight ones like FPN and MLP-decoder.
VARYING WINDOW ATTENTION
3.1 PRELIMINARY: LOCAL WINDOW ATTENTION
Local window attention (LWA) is an efficient variant of Multi-Head Self-Attention (MHSA), as shown in Fig. 2.. Assuming the input a 2D feature map denoted as X2dd E R. Cx W the first step is reshaping it to local windows, which can be formulated by:
X2d Unfold (kernel P. stride P) (x2a),
where Unfold() is a Pytorch Paszke et al., 2019) function (See Pytorch official website for more information). Then the MHSA operates only within the local window instead of the whole feature.
3

---

Published as a conference paper at ICLR 2024
To show the efficiency of local window attention, we list its computation cost to compare with that of MHSA on the global feature (Global Attention):
Ω (GA) = ((WW)CC C² + 2(HW))
Ω (LWA) = ((W) 2(H (2)
Note that the first term is linear mappings, Le., query, key, value, and out, and the second term on the attention computation, i.e., calculation of attention matrices and the reighted-summattee of value. In the high-dimensional feature space, p2 is smaller than C' and much smaller than HW. Therefore, the cost of attention computation in LWA much smaller than the cost of linear mappings which is much smaller than the cost of attention computation in GA.
Besides, the memory footprints of GA and LWA are listed below, showing the of LWA. The intermediate outputs of the attention mechanism involve query, key, value, and out, all of which are outputs of linear mappings, and attention matrices output from attention computation.
Mem. (GA) (HW)C + (HW)²
Mem. (LWA) x (HW)C + (HW)P?
(3)
The consequence of the computational comparison remains valid. GA the second term is much larger than the first, but in LWA the second term is smaller than the first.
3.2 VARYING THE CONTEXT WINDOW
In LWA, &2a output by Eq. will attend to itself. VWA, the query is still 82a, but for the context, by denoting it Czd. the generation can be formulated as:
C2d = Unfold (kernel = RP, stride = P, padding = zero) (x2a)
(4)
From the view of window sliding, the query generation is P P window with stride of P x P sliding on X2a, and the context generation is a larger RP x RP window with still a stride of P x P sliding X2d- R is the varying ratio, constant value in one VWA. As shown in Fig. 2 when R is 1, VWA becomes LWA, and the query and context are entangled together in the local window. But when R 1, with the enlargement of context, the query can see wider than the field of the local window. Thus, VWA is a variant of LWA and LWA is a special case of VWA, where R in VWA.
From the illustration of Fig. bb the computation cost of VWA be computed by:
Ω (VWA) 2 1002++1((100) () 2HH))
(5)
2 the extra computation cost caused by enlarging the context patch is Subtracting Eq. 5 from Eq. quantified:
Ω (EX.) = -1
()WW)
(6)
For the memory footprint of VWA, can be computed by according Fig. Pp:
Mem. (VWA) (HW))C (HW)
Subtracting Eq. 7 from Eq. 3 the extra memory footprint is:
Mem. (EX.) x (R² (HW) + + (R²
(8)
Apparently, the larger the window, the more challenging the problem becomes. First, the efficiency advantage of attention computation (the second term) LWA does not hold. Second, linear map- pings, the first term, yield much more computation budget, which is more challenging because to our knowledge existing works on making attention mechanisms efficient rarely take effort reduce both the computation cost and memory footprint of linear mappings and their mapping outputs. Next, we will introduce how to address the dilemma caused by varying the context window.
3.3 ELIMINATING EXTRA COSTS
With the analysis of Eq. 6 and Eq. 8 the most straightforward way to eliminate the extra cost and memory footprint is re- re-scaling the large context E IR Cx Rx Px Rx back to the same size as that of
the local query R Cx Px which means R is set to 1 and thereby both of Eq. 6 and Eq. g O.
Above all, it is necessary to clarify the difference between using this idea to deal with the extra computation cost and the extra memory footprint. As shown in Fig. 2b, the intermediate produced
4

---

Published as a conference paper at ICLR 2024
sliding stride
sliding stride
sliding stride
sliding stride
CRR
local window
K
RP
K
DOPE
RP
C/R²
K
window
V
context
PE
RP
(a) local window attention
(b) varying window attention (post-scaling)
(c) varying window attention (pre-scaling w/ DOPE)
Figure 2: (a) illustrates that in LWA, Q. K, and V are all transformed from the local window. (b) illustrates naive implementation of VWA. Q is transformed from the local window. and V are re-scaled from the varing window. PE is short for Patch Embedding. R (of RP) denotes the size ratio of the context window to the local window (query). (c) illustrates the professional implementation of VWA. DOPE is short for densely-overllppinng Patch Embedding.
by varying (enlarging) the window, which is the output of Eq. 4 already takes the memory that is R HW)C. Therefore, re-scaling the large context after generating it does not work, the right step should be re-scaling the feature X2d before running Eq. 4 We name this pre-scaling principle.
Solving the problem begun by the pre-scaling principle. A new feature scaling paradigm, densely overlapping patch embedding (DOPE), is proposed. This method is different from patch embedding (PE) widely applied in ViT and HVT as it does not change the spatial dimension but only changes the dimensionality. Specifically, for X2d, after applying Eq. oon it, the output's shape is:
H/P x W/P x RP X RP C.
(9)
which produces the memory footprint of R R2W.C. C. Instead, DOPE first reduces the dimensionality of X2d from C to C/ C²RR,, and then applies Eq. 4 resulting in the context with shape of:
H/P x W/P x RP x RP x C/R².
(10)
which produces the memory footprint of HWC, the same as X2d, eliminating the extra memory.
Since PE is often implemented using conv layers, how DOPE re-scales features expressed as:
DOPE Conv2d(in C, C/R², kernel R, stride 1).
(11)
So, the term "densely overlapping" of DOPE describes the densely arranged pattern of convolutional
kernels, especially when R is large, filtering every position. The computation cost introduced by DOPE can be computed by:
Ω (DOPE) = R x R C C/R² x HW = CH
(12)
This is equivalent to the computation budget required for just one linear mapping.
However, the context window E RRRXPPXC/ processed by DOPE cannot be attended to by the query window E PxPC We choose PE downsample the context and increase its dimension- ality to a new context window E I Pxxxx The PE function can be formulated as:
PE = Conv2d(in = C/R², out = C, kernel = R, stride = R.).
(13)
The computation cost for one context window applying PE is:
Ω (PE for one context) = R R C/R² C RP/RX RP/R=
(14)
For all context windows from DOPE, with total of H/P x W/P, the computation cost becomes:
Ω (PE) H/P w/P x Ω (PE for one context) (HW)C?.
(15)
This is still the same as only one linear mapping.
After applying the re-scaling strategy described, as shown in Fig. 2 it is clear that the memory footprint of VWA is the same as Mem. (LWA) in Eq. 3 not affected by the context enlargement. The attention computation cost is also the same as ((TWA) in Eq. 2 For DOPE, VWA uses it once, thus adding one linear mapping computation to (LTWA). For PE, VWA uses it twice for mapping the key and value from the DOPE's output, replacing the original key and value mapping. So the computation cost of VWA merely increases 25% one linear mapping of (HI)) than LWA:
Ω (VWA)= (4 H) 222.
(16)
5

---

Published as a conference paper at ICLR 2024
stin. mus
copy&ssiff
top pad.
chif
map
altn map
bottom pod.
map
(8) attention collapse
propoced padding mode
(c) CSP shifts windows towards feutures
Figure 3: (a) illustrates the zero-padding mode caused attention collapse when the context window is
very large and the context window surrounds the local window near the corner or edge. (b) illustrates the proposed copy-shift padding (CSP) mode. The color change indicates where the padding pixels are from. (c) CSP is equivalent to moving the context windows towards the feature, ensuring that every pixel the query attends has different valid non-zero value. Best viewed in color.
3.4 ATTENTION COLLAPSE AND COPY-SHIFT PADDING
The padding mode in Eq. is zero padding. However, visualizing attention maps of VWA, we find that the attention weights of the context window the corner and edge tend to have the same value, which makes attention collapse. The reason is that too many same zeros lead to smoothing the probability distribution during Softmax activation. As shown in Fig. 3 to address this problem, we propose copy-shift padding (CSP) equivalent to making the coverage of the large window move towards the feature. Specifically, for the left and right edges, X2a after CSP
X2d = Concat(d = (x2all....,
(rsall...(45 (R+ 1)P/2: RP), X2d. X2al..., - RP - 1)P/2)).
(17)
where Concat() denotes the Pytorch function concatenating tuple of features along the dimension d. Based on X2d obtained by Eq. 17 CSP padding the top and bottom sides can be formulated by:
X2d
Concat(d ()2a .... (R 1)P/2: RP, :l, X21; X2d(..., RRP (R P/2,): (18)
Hierarchical Backbone
VWFormer
Short
Intor Poeropoae
Path
Stage
MLP
Fi E For 2256
VIWA
F1.
mm
F
MLD Interpoaut
VWA
F F
2G12
MPP
VWV
MLPea
multi-layer aggregation
multi-scale representations
low-leyel enhancement
Figure 4: VWFormer contains multi-layer aggregation, learning multi-scale representations, and low-level enhancement. Like other MSDs, VWFormer takes multi-level feature maps as inputs.
VWFORMER
Multi-I Aggregation As illustrated in Fig. VWFormer first concatenates feature maps from the last three stages instead of all four levels for efficiency, by upsampling the last two (Fis and F32) both to the same size the 2nd-stage one (Fs), and then transform the concatenation with one linear layer (MLP)) to reduce the channel number, with F as the outcome.
Multi-Scale Representations To learn multi-scale representations, three VWA mechanisms with varying ratios 2, 4, 8 are paralleled to act on the multi-layer aggregation's 's output F. The local window size P of every VWA is set to H W. path, exactly a linear mapping layer, consummates the very local scale. The MI.Ps of VWFormer consist of two layers. The first layer (MLP,) is linear reduction of multi-scale representations.
8 8 subject to the spatial size of F. Additionally, the short
Low-Level Enhancement The second layer (MLP2) of MLPss empowers the output Fi of the first layer with low-level enhancement (LLE). LLE first uses linear layer (MLP with small output channel numbers 48 to reduce the lowest-level F, dimensionality. Then F, is upsampled to the same size as MLP output From and fused with it through MT.P,, outputting F2.

---

Published as a conference paper at ICLR 2024
5 EXPERIMENTS
5.1 DATASET AND IMPLEMENTATION
Experiments conducted three public datasets including Cityscapes, ADE20K, and cOcoStuff-164K (See D.2 for more information). The experiment protocols are the same as the compared method's official repository. For ablation studies, we choose the Swin-Base backbone as the testbed and use the same protocols as Swin- Crennet (See D.3 for more information).
5.2
MAIN RESULTS
5.2.1
COMPARISON WITH SEGFORMER (MLP-DECODER)
SegFormer uses MixFormer (MiT) as the backbone and designs a lightweight MIP--dcoddrr as MSD decode multi-scale representations of MixFormer. To demonstrate the effectiveness of Woromer in improving multi-scale representations by VWA, we replace the MI.P-decoder SerForrr with VWFormer. T the number of parameters, FLOPs, memory footprints, and mloU. Across all variants of backbone MīT (B0-+5)), VWFormer trumps MLP-decoder on every metric.
Table 1: Comparison of SegFormer (MiT-MLP) with VW-SegFormer (MiT-VW.).
ADE20K
Cityscapes
COCO
MSD
backbone
Params(M)
FLOPs(G) 1
mem.(G)
mloU(MS)
mloU(/MS)
mloUf
MIT-BO
3.8
37.4/ 38.0
76.2
35.6
MiT-B1
13.7
15.9
2.7
42.2/
78.5 80.0
40.2
MîT-B2
27.5
62.4
4.3
46.5 47.5 47.55
81.0/ 82.2
44,6
MLP
MiT-B3
47.3
79.0
5.6
49.4 50.0
81.7 / 83.3
45.5
MiT-B4
64.1
95.7
50.3 / 51.1
81.9 83.4
46.5
MIT-B5
84.7
112
8.1
51.0/ 51.8
82.3 / 83.5
46.7
MIT-BO
3.7
5.8
2.1
38.9/39.6
77.2/
36.2
MIT-BI
13.7
43.2 44.0
79.0/ 80.4
41.5
MîT-B2
27.4
46.6
4.3
48.1 1..1/99
81.7 / 82.7
45.2
VW.
MIT-B3
47.3
63.3
5.6
50.3 50.9
82.4/ 83.6
46.8
MiT-B4
64.0
79.9
7.0
50.8 51.6
82.7 / 84.0
47.6
MiT-BS
84.6
96.1
8.1
52.0/
82.8 84.3
48.0
5.2.2
COMPARISON WITH UPERNET
In recent research, UperNet was often used as MSD evaluate the proposed vision backbone in semantic segmentation. Before multi-scale fusion, UperNet learns multi-scale representations by utilizing PSPNet (with scale inadequacy issue) merely on the highest-level feature map. In contrast, VWFormer rectify ERFs of every fused multi-level feature map in advance. Table shows VWFormer consistently uses much fewer budgets to achieve higher performance.
Table 2: Comparison of UperNet with VWFormer. Swin Transformer and ConvNeXt serve as backbones. VW-Wide is VWFormer with two times larger channels.
ADE20K
Cityscapes mloU(/MS)
MSD
backbone
params(M)
FLOPs(G) 4
(G)
mloU(/MS)
Swin-B
120
306
8.7
50.8 / 52.4
82.3 / 82.9
Swin-L
232
420
12.7
52.1 53.5
82.8 83.3
UperNet
ConvNeXi-B
121
293
5.8
52.1 / 52.7
82.6/ / 82,9
ConvNeXi-L
233
394
8.9
53.2 53.4
83.0 / 83.5
ConvNeXi-XL
389
534
12.8
53.6 / 54.1
83.1 / 83.5
Swin-B
95
120
7.6
52.5 / 53.5
82.7 / 83.3
Swin-I.
202
236
11.5
54.2
83.2 83.9
VW.
ConvNeXt-B
83.9
ConvNeXt-L
205
208
54.3 55.1
83.5 84.1
ConvNeXt-XL
357
346
11.4
54.6 55.3
83.6 84.3
WW-Wide
Swin-L
223
306
13.7
54.7 56.0
83.5 / 84.2

---

MaskFormer and Published as a conference paper at ICLR 2024
5.2.3 COMPARISON WITH MASKFORMER AND MASK2FORMER
MaskFormer and Mask2Former introduce the mask classification mechanism for image segmenta- tion but also rely MSDs. MaskFormer uses the FPN MSD. while Mask2Former empowers multi-level feature maps with feature interaction by integrating Deformable Attention (Zhu et al, 2020) into FPN. Table 3 demonstrates that VWFormer is efficient FPN and achieves mloU gains from 0.8% to 1.7%. The results also show that VWFormer performs stronger than Deformable Attention with less computation costs. The combo of VWFormer and Deformable Attention further improves mloU by 0.7%-1.4% This demonstrates VWFormer can still boost the performance of interacted multi-level feature maps via Deformable Attention, highlighting its generability.
Table 3: Comparison of VWFormer with FPN and Deformable Attention.
Fask2Former serve as testheds (mask classification heads).
head
MSD
backhone
params(M) 4
FLOPs(G)
mem.(G))
mloU(/MS)
Swin-T
41.8
57.3
46.7
Swin-S
63.1
81.1
5.5
49.4 51.0
FPN
Swin-B
102
126
8.2
52.7 / 53.9
Swin-L
212
239
11.5
54.1 55.6
MaskFormer
Swin-T
42.8
55.6
5.3
47.8/49.0
Swin-S
64.1
79.4
6.2
50.5 52.7
VW.
Swin-B
102
124
8.5
53.8 / 54.6
Swin-T.
213
237
12.0
55.3 / 56.5
Swin-T
47.4
74.1
5.4
47.7 1.77
Swin-S
68.8
97.9
6.6
51.3 52.4
Deform-Attn
Swin-B
107
142
9.9
53.9 55.1
Swin-L
215
255
56.1/
Swin-T
47.7
60.8
3.6
50.3
Swin-S
84.6
4.7
52.1 53.7
Mask2Former
VW.
Swin-B
108
129
7.9
$4.6 56.0
Swin-L
217
244
12.1
56.5 / 57.8
Swin-T
53.3
85.3
6.2
48.5 50.4
Swin-S
74.7
108
7.3
52.0/ 53.6
VW.
Swin-B
113
152
10.1
55.2 56.5
(Deform-Att))
Swin-L
221
266
13.7
56.8 / 58.3
5.3 ABLATION STUDIES
5.3.1 SCALE CONTRIBUTION
4show the performance drops when removing any VWA of VWFormer. These results indi- cate every scale is crucial, suggesting that scale inadequacy is fatal to multi-scal learning. Also, we
Table 4 shows
add VWA branch with R context windows which is exactly LWA, and then substitute R VWA with it. The results show LWA unnecessary in VWFormer because the short path (1 x convolution) in VWFormer can provide a very local receptive field, as visualized in Fig.
Table 4: Performance of different Scale combinations. Conducted ADE20K. The numbers of
"scale group" are varying ratios. (2, 4. 8) is the default setting.
backbone scale group
Swin-B
4, 8)
(2, 4)
(2, 8)
(4, 8)
(1, 2, 8)
(1, (1,4. 51.9 52.9
mloU(/MS)t
52.5/53.5
51.8/ 52.8
52.7
51.9 52.8
52.1 / 53.0
5.3.2 PRE-SCALING VS. POST-SCALING
Table compares: applying VWA without rescaling, with naive rescaling as depicted in Fig. 2p. and our proposed professional strategy. VWA originally consumes unaffordable FLOPs and mem- ory footprints. Applying the naive scaling strategy saves some FLOPs and memory footprints, but introduces patch embedding (PE) increasing amount of parameters. Our proposed strategy does not only climinate the computation and memory introduced by varying the context window but also only adds small number of parameters. Moreover, it does not sacrifice performance for efficiency.
8

---

Published as a conference paper at ICLR 2024
Table 5: Performance of different ways to re-scale the context window. Conducted on ADE20K.
backbone
post.
rescaling method
params.4
FLOPs(G)
mem.(G))
mloU(/MS)f
pre-scaling post-scaling
DOPE + PE
95
120
7.6
53.5/53.5
PE
115
210
9.9
52.3 53.6
Swin-B
post-scaling
Avg. Pooling
93
114
9.3
51.7 52.8
rescaling
93
315
13.1
52.4/53.4
5.3.3 ZERO PADDING VS. VW PADDING
The left table of Table 6 shows using zero padding to obtain the context window results in a 0.8% lower mloU than applying CSP to obtaín the context window. Such a performance loss is as severe as removing one scale of VWA, demonstrating the harm of attention collapse and the necessity of our proposed CSP in applying the varying window scheme.
Table 6: Left: Performance of zero padding mode and our proposed CSP. Right: Performance of different output channel number settings of LLE module in VWFormer.
backbone
Swin-B B
backbone
Swin-B LLE 52.5 120
padding
zero
CSP
method
FPN
mloU(/MS)
52.0/52.7
52.5 53.5
mloU /LLOP)) 51.8/112
52.1 176
5.3.4 EFFECTIVENESS OF LOW-LEVEL ENHANCEMENT
The right table of Table 6 analyzes Low-Level Enhancement (LLE). First, removing LLE degrades mloU by 0.7%. From Fig. L it can be seen that the lowest-level feature map is of unique receptivity, very local global, adding new scales VWFormer's multi-scale learning. FPN is also evaluated as an alternative, and the results show FPN is neither stronger nor cheaper than LLE.
Image
resllt
VWW result
Image
rosilt
VW. resllt
Image
Seg. reult
VW result
Sez: ETR
EFR
GT
Seg.
VW. EFR
GT
Sep. EFR
(a)
(b)
(c)
Figure 5: Visualization of inference results and ERFs of SegFormer and VWFormer. The red dot is the query location. The red box exhibits our method's 's receptive superiority. Zoom in to see details.
SPECIFIC ERF VISUALIZATION
The ERF visualization Fig is averaged on many ADE20k val images. To further substanti- ate the proposed issue, Fig. analyzes the specific ADE20K val image with ERFs of segformer and VWFormer contrastively. This new visualization can help to understand the receptive issue of existing multi-scale representations and show the strengths of VWFormer's multi-scale learning.
Fig. 5a showcases a waterfall along with rocks, Our VWFormer's result labels most of the rocks, but SegFormer's result struggles distinguish between "rock" and mountain". From their ERFs. it can be contrastively revealed that VWFormer helps the query to understand the complex scene, even delineating the waterfall and rocks, more distinctly than SegFormer within the whole image.
Fig. 5p showcases meeting room with table surrounded by swivel chairs. Our VWFomeer's result labels all of the swivel chairs, but SegFormer's result mistakes two swivel chairs as general chairs. From their ERFs, it can be contrastively revealed when VWFormer infers the location, it incorporates the context of swivel chairs, within the Red box on the opposite side of the table. But SegFormer neglects to learn about that contextual information due to its scale issues.
Fig. 5p showcases white tall building. Our VWFormer's result labels correctly, but SegFormer's result mistakes part of the building as the class "house". From their ERFs, it can be contrastively re- vealed that VWFormer has clearer receptivity than SegFormer within the Red box which indicates this object is church-style building.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was supported by the National Natural Science Foundation of China (NSFC) under Grant 62076093.
REFERENCES
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1209 1218, 2018.
Liang-Chieh Chng-Chhehhhh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint vrXiv:1706.05560 2017.
Liang-Chies Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801-818, 2018.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint wwXiv22255.08544 2022.
Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34:17864-17875, 2021.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1290-1299, 2022.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler. Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Jiaqi Gu, Hyoukjun Kwon, Dilin Wang. Wei Ye, Meng Li, Yu-Hsin Chen, Liangzhen Lai, Vikas Chandra, and David Pan. Multi-scale high-resolution vision transformer for semantic segmen- tation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12094-12103, 2022.
Junjun He, Zhongying Deng, and Yu Qiao. Dynamic multi-scale filters for semantic segmentation. In Proceedings the IEEF/CVF International Conference Computer Vision, pp. 3562-3572, 2019a.
Junjun He, Zhongying Deng, Lei Zhou, Yali Wang, and Yu Qiao. Adaptive pyramid context network for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7519-7528, 2019b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Shihua Huang, Zhichao Lu, Ran Cheng, and Cheng He. Fapn: Feature-aligned pyramid network for dense image prediction. In Proceedings of the IEEE/CVF international conference computer vision, pp. 864-873 873,873,777 2021.
Alexander Kirillov, Ross Girshick. Kaiming He, and Piotr Dollár. Panoptic feature pyramid net- works. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6399-6408, 2019.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE conference on com- puler vision and pattern recognition, pp. 2117-2125. 2017.
10

---

Published as a conference paper at ICLR 2024
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer. Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-11986 2022.
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field deep convolutional neural networks. Advances in neural information processing systems, 29, 2016.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Sucheng Ren, Daquan Zhou, Shengfeng He, Jiashi Feng, and Xinchao Wang. Shunted self-attention via multi-scale token aggregation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10853-10862, 2022.
Tete Xiao, Yingeheng Liu, Bolci Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings the European conference on computer vision (ECCV), pp- 418 434, 2018
Enze Xie, Wenhai Wang. Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34: 34:12077-12090. 2021.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. arXiv preprint arXiv:2107.00641, 2021.
Maoke Yang, Kun Yu, Chi Zhang, Zhiwei Li, and Kuiyuan Yang. Denseaspp for semantic segmen- tation in street scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2018
Qihang Yu, Yingda Xia, Yutong Bai, Yongyi Lu, Alan L Yuille, and Wei Shen. Glance-and-gaze vision transformer. Advances in Neural Information Processing Systems, 34: 19922-139030,, 2021.
Yuhui Yuan, Lang Huang, Jianyuan Guo, Chao Zhang, Xilin Chen, and Jingdong Wang. Ocnet: Object context network for scene parsing. arXiv preprint weXiv:1809.009// 2018.
Bowen Zhang, Liyang Liu, Minh Hieu Phan, Zhi Tian, Chunhua Shen, and Yifan Liu. Segvitv2: Exploring efficient and continual semantic segmentation with plain vision transformers. arXiv preprint arXiv:2306. 06289, 2023.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2881-2890, 2017.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633 633-641, 2017.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
Zhen Zhu, Mengde Xu, Song Bai, Tengteng Huang, and Xiang Bai. Asymmetric non-local neural networks for semantic segmentation. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 593 602, 2019.
11

---

Published as a conference paper at ICLR 2024
ASPP: rates from left right (1, 12, 24, 36)
(b) PSP: output from left right (1, 3,
(c) ConvNeXt: stagel, stage2, stage3, stage4
(d) Swin Transformer. stagel, stage2, stage3, stago4
(c) SogFormer: stagel, stage2, stage3, stage4
(i) proposed VWA: residual path, R-2,
Figure 6: ERF visualization multi-scale representations learned by (a) ASPP, (b) PSP, (c) Con- vNeXt, (d) Swin Transformer, (e) SegFormer, and (f) Our proposed varying window atfention. ERF maps are visualized across 100 images of ADE20K validation set. This figure exactly Fig.
A QUALITATIVE ANALYSIS OF TYPICAL METHODS' ERFS
Below is detailed analysis of the issues with methods visualized in Fig. For good readability, Fig. copied and pasted here as Fig.
ASPP employs atrous convs with set of reasonable fixed atrous rates learn multi-scale repre- sentations. However, as shown in Fig. [o. the largest receptive field does not capture the desired scale of representations. This because the parameter settings are manual and do not adapt the image size. The lack of adaptability becomes more severe when training and testing samples have different sizes, common occurrence with applying strategies like test-time augmentation (TTA). Furthermore, when the receptive field is large, contributions from the atrous parts are zero, leading to inactivated subareas within larger receptive fields.
PSP applies pooling filters with different scales by adjusting the hyper-parameter, output size of adaptive pooling, to learn multi-scale representations. However, as shown Fig. (b, the receptive field sizes exactly the same for output sizes 1 and 2 and for output sizes 3 and 6. This is because the super small output needs to be interpolated to the original feature size. During the interpolation, if a position does require interpolation obtain its value, its receptive field remains unchanged. However, interpolation needed, the receptive field can be influenced by other positions.
ConvNeXt stages' receptive field sizes change from small to large the network deepens. This is because the stacking of multiple 7x7 convolutions can simulate much larger convolutional kernels. However, shown in Fig. G6. compared ASPP and PSP, the largest receptive field of the four scales in ConvNeXt only covers half of the original image and does not capture a global represen- tation because the 7x7 conv is still of locality. Additionally, it is hard to distinguish between the receptive field scale of the third and the fourth stage.
Swin Transformer's basic layers consist of local window attention mechanisms and shift-window attention mechanisms. The feature maps in its four stages exhibit an increase in receptive field size from small large. Swin Transformer also faces challenges learning global representations effectively. Moreover, due to the shift operation of the local window, its receptive field shape is irregular as shown in Fig. Gdt leading inactivated subareas within the receptive field range.
SegFormer's basic layers are sophisticated, incorporating local window attention, global pooling attention, and 3x3 convolutions. It is hence difficult to imagine the receptive field shape and size for its four-level feature maps. Fig. 6 indicates that SegFormer learns global representations in the low-level layers (i.e., the first and second levels) but still suffers from inactivated subareas within the receptive field range. In the higher layers (i.e., the third and fourth levels), they learn more localized representations but their field ranges are very similar. Therefore SegFormer also meets the scale inadequacy because it can only learn global and local representations.
12

---

Published as a conference paper at ICLR 2024
(a) ISANet
(b)
Figure 7: (a) contains ERF maps two-scale (left is global and right is local) representations learnt by ISANet. (b) is the ERF map of global representation learnt by ANN
B ERFs OF EXISTING MULTI-SCALE ATTENTION (RELATIONAL
MULTI-SCALE LEARNER)
Fig. 7 visualizes the ERF map of ISANet (Yuan et al., 2018), merely learning local and global representations while ignoring other scales. So the issue of scale inadequacy for ISANet very clear. The local representation is learned using the local window attention mechanism, while the global representation is obtained by interlacing pixels from all windows to create new windows that contain pixels from each original local window. Then, the window attention mechanism is applied to the new window. The ERF map shows that their receptive fields are not continuous due to interlacing, suggesting that ISANet also meets field inactivation.
ANN Zhu al. (2019) adaptive pooling capture multi-scale features in PSP manner. Then they are together attended to by the original feature which serves the query. The scale of the receptive field is singly global because every context filtered by adaptive pooling derived from the whole feature map. So the issue of scale inadequacy is also very clear for ANN. Fig. 7 shows the activation does not spread the global range uniformly and the bottom area insufficiently activated. Therefore, both scale inadequacy and field inactivation are issues of ANN and its relevant methods.
empirically compare ours to ISANet ANN, VWFormer out- performs both of them by large margins consistently across different backbones and benchmarks.
The bottom three rows of Table 7
Table 7: Comparison of VWFormer with other receptive-field-varribbl multi-scale learners. Red, Green, Blue highlight the top-3 results of one metric.
ADE20K
Cityscapes
101
ResNet50
ResNet101
ResNet50
ResNet101
Method
L
FLOPs(G))
mloU(/MS)
mloU(/MS)t
mloU(/MS)
mloU(/MS))
PSPNet 68.0
254.9
41.1/42.0
43.6/44. 44.4
77.9/79.2
79.0 80.0
D3epLabV3
87.1
346.1
42.4 43.3
44.0 45.2
80.7
79.7 / 80.8
DeepLabV3+
62.6
252.8
42.7//33
44.6/46.0
79.9 81.0
81.07 82.2
APCNet
75.4
282.0
42.2/43.3
45.5
78.8 / 80.0
79.7 / 80.6
DMNet
273.4
42.5 43..6
45.3
79.2 80.2
79.6/
DenseASPP
96.8
461.2
43.5/43.5
43.8 44.9
79.6 80.4
80.3 / 81.0
ANN
65.2
262.6
41.0//2.3
43.0/44.2
78.9 80.6
78.8 / 80.4
ISANet
53.7
227.9
41.1 42.4
42.6/43.1
79.3 / 80.5
80.6 / 81.6
VWFormer
50.4
203.2
43.5 44.4
45.9 47.0
80.3 81.2
81.5 82.7
C
MORE EXPERIMENTAL ANALYSES
C.1 COMPARISON OF VWFORMER WITH MULTI-SCALE LEARNERS
To verify the superiority of V WWromer over representative multi-scale learners for semantic segmen- tation, Table 7 compares VWFormer with PSPNet /Zhao 7SPNetZZhao et al. (017)). a
Deepl | Chen al. (017). DecpLabV3+ Chen et al. (2018), DenseASPP Yang et al. (2018), APCNet He et al. (2019b). DM- Net et al. 2019a), ANN Zhu et al. (2019). and ISANet Yuan et al. (2018). For fairness, we em- ploy the same backbones for all the methods, including ResNet50 and ResNet101 He et al. 2016 All methods are trained for 80000 iterations, and evaluated on Cityscapes as well as ADE20K. The input size is 768/76 769 x 768/70 769 for Cityscapes, and 512 x 512 for ADE20K.
From Table 7 we can find that VWFormer brings the best results to both ResNet50 and ResNet101 on both datasets. Specifically, Cityscapes, VWFormer achieves 81 31.2% mloU with ResNet50,
13

---

Published as a conference paper at ICLR 2024
and 82.7% mloU with ResNet101, which the best results among all methods. DeepLab V3+ achieves the closest performance to ours but has more computation costs and parameters by 49. 19.6G and 12.2M, respectively. On ADE20K, VWFormer outperforms other methods by large margins consistently. APCNet performs most closely to ours, but VWFormer uses the least FLOPs and parameters. In short word, VWFormer is more powerful than any other multi-scale learners.
C.2 VWFORMER WITH SOTA METHODS
Table analyzes method briefly with state-of-the-art semantic segmentation methods created on other tracks. Left of Table 8 shows the comparison with HRViT [u et al. (2022), which is hierarchical Vision Transformer (HVT) with complex multi-scale learning. was paired originally with MLP-decoder from SegFormer as MSD. Moreover, Former. The performance gains are considerable, supporting VWFormer's capability of improving multi-scale representations.
is replaced with our VW-
Center compares VWFormer with SegViT-V2 Zhang al. (2023). V2 a decoder specif- ically for ViT (or categorized plain Vision Transformer). Here VWFormer cooperates with plain Vision Transformer at the first time. The improvement shows that VWFormer not only effective in HVT but also powerful in plain backbone architecture.
Right shows the comparison with ViT-Adapter Chen et al. (2022), which is pre-training technique for improving VIT on dense prediction tasks. Like many works on Vision Transformer employing UperNet as MSD for semantic segmentation, ViT-Adapter was also originally paired with UperNet. Moreover, UperNet is replaced with VWFormer, achieving considerable performance gains.
Table Left: VWFormer paired with HRViT for comparison with original HRViT. Center: Com-
parison of VWFormer with V22 Right: VWFormer paired with Adapter for comparison with original Adapter (paired with UperNet). Evaluated on ADE20K with multi-scale inference.
mIoU
HRViT-b1
b2
b3
mloU
BEiT-V2-L
mloU
Ada.-B
Ada.-
MLP
45.6
48.8
50.2
SegViT-V2 VW.
58.2
Uper. 52.5
54.4
VW.
46.9
50.0
51.6
58.8
VW.
53.5
55.2
C.3 EXAMINING INFERENCE TIME
Table 9 shows supplementary results of inference time for Table Table 2 and Table From Top of Table g VWFormer's s inference time is faster than SegFormer. From Bottom Left, VWFormer's inference time is faster than UperNet. From Bottom Right, VWFormer's inference time is faster than FPN. Additionally, by comparing the results the last row of Bottom Left and the first row of Bottom Right, it can be observed that VWFormer is faster than MaskFormer.
Table Top: Frames/Sec.de (FPS) of VWFormer and SegFormer. Bottom Left: FPS VWFormer and UperNet. Bottom Right: FPS of VWFormer and MaskFormer. Evaluated 512 x 512 crop for MIT and Swin- (Ti and S). Evaluated on 640 x 640 crop for ConvNeXt/ Swin-(B, L, and XL)
FPS 1
MiT-BO
MiT-B1
MrT-B2
MiT-B3
MiT-B4
MiT-BS
SegFormer
30.5
28.9
20.6
16.9
14.3
12.0
VWFormer
30.8
29.4
21.1
17.4
14.6
12.5
FPS
Swin-B
L
ConvNeXt-B
XL
FPS f
Swin-Ti
S
B
L
Uper.
9.9
7.7
15.8
13.5
11.1
Mask.-FPN
22.1
19.6
11.6
7.9
VW.
12.0
9.0
19.7
17.2
14.6
Mask--WW
23.1
20.7
11.9
7.9
C.4
BREAKDOWN OF PERFORMANCE GAINS
Table shows breakdown of performance gains within Cityscapes which has 19-class segments. The upper results are obtained by MiT-B5 paired with SegFormer head (mloU 82.26%) and the lower results obtained by MiT-B5 paired with our VWFormer (mloU 82.87%).
The bold number is the class that the counterpart performs better than ours. Except for the "truck" largely, which seems like biased result, the 'wall',
class where SegFormer outperforms 'sky", and 'train' SegFormer only slightly outperforms ours (by avg. 0.2%). And on the other 15 classes, Ours shows consistent superiority SegFormer (by avg. 1.4%).
14

---

Published as a conference paper at ICLR 2024
Table 10: Top: Nine classes performance comparison of SegFormer and VWFormer on Cityscapes.
Bottom: Ten classes performance comparison of SegFormer and VWFormer on Cityscapes.
IoU
road
sidewalk
building
wall
fence
pole
light
sign
vegetation
Seg.
98.5
87.3
93.7
68.6
65.7
69.5
75.6
81.8
93.2
VW.
98.5
87.6
94.0
68.4
68.7
73.0
77.3
84.5
93.5
IoU
terrain
sky
person
rider
car
truck
bus
train
motorbike
bicycle
Seg.
66.2
95.7
85.3
69.6
95.6
85.7
91.7
84.7
73.8
80.7
VW.
66.3
95.4
86.8
71.2
96.2
77.4
93.1
84.6
75.2
82.2
D
SOME DETAILS
D.1 DETAILS OF VWFORMER CAPACITY SETTING
Sec. and Fig. indicate the flow of channel numbers is 512(output of multi-layer aggregation) (004((cncateeation of learnt multi-scale representations)- 512(utt of multi- scale aggregation) 512 + 48 5600concatenation of LLE) 256(final output of VWFormer).
For some lightweight backbones, such channel settings incur too much computational burden. We further introduce an efficient setting for VWFormer to cooperate with lightweight backbones such as SegFormer-BO and SegFormer-Bl. The new flow of channels is 128 (output of multi-layer aggregation) 512(concatenation of learned multi-scale representations)- 128(output of multi- scale aggregation) 128 + 32 160(concatenation of LLE) 128(final output of VWFormer)
D.2 DETAILS OF DATASET
Cityscapes an urban scene parsing dataset that contains 5, 000 fine-annotated images captured from 50 cities with 19 semantic classes. There 2, 9.5 images divided into training set, 500 images divided into a validation set, and 1, 525 images divided into testing set.
ADE20K is challenging dataset in with 150 categories, a testing set of 3, 352 images, and a validation set of 2, 000 images.
parsing. It consists of training of 20, 210 images
COCOStuff-1666 is very challenging benchmark. It consists of 164k images with 171 semantic classes. The training set contains 118k images, the test-dev dataset contains 20K images and the validation set contains 5k images.
D.3 DETAILS IMPLEMENTATION
Experiments of comparison with SegFormer and UperNet are implemented based on the MM- Segmentation codebase. addition, ablation studies are done with MMSegmentation. Experiments comparing with MaskFormer and Mask2Former are implemented based on the Detectron2 code- base The computing server on which all experiments are run has 16 Tesla V100 GPU cards. For other methods' results, report the number shown in their papers.
E QUALITATIVE RESULTS
As shown in Fig. and Fig. present qualitative results ADE20K of SegFormer and VWFormer with MiT-B5 as the backbone. The yellow dotted box focuses on the apparent visu- alization difference between them and the Ground Truth (GT). Compared SegFormer's results, VWFormer improves the inner consistency of objects. Taking the bedroom (the first row shown Fig 8 an example, part of the shelf that is near the bed is misidentified the shelf by Seg- Former, and the boundary between the bed and shelf is extremely unclear. In contrast, VWFormer segments the two objects very finely, which provides coherent boundary. Moreover, we observe that with VWFormer similar objects are hardly confused. For example, in the living room shown in the row of Fig. 9 SegFormer mistakes the sofa for an armchair. And the first row of Fig. 9 SegFormer mistakes the blind for windowpanes. However, VWFormer accurately distinguishes be- tween the sofa and the armchair, as well as between the blinds and the windowpanes.
15

---

Published as a conference paper at ICLR 2024
Image
SegFormer
VWFormer
GT
Figure 8: Qualitative results of ADE20K validation set. MiT-B5 serves as the backbone
16

---

Published as a conference paper at ICLR 2024
Image
SegFormer
VWFormer
GT
Figure 9: Qualitative results of ADE20K validation set. MiT-B5 serves as the backbone.
17


Reviews:
[{"review_id": "aSXuhTnBLa", "reviewer": "Reviewer_1jxp", "summary": "The submission introduces Varying Window Attention (VWA) to address scale inadequacy and field inactivation in semantic segmentation. VWA modifies the local window attention mechanism to dynamically adjust the receptive field, aiming to improve multi-scale representation learning. The paper also proposes a new multi-scale decoder, VWFormer, which incorporates VWA and demonstrates performance and efficiency gains on standard datasets. The work claims three main contributions: the VWA mechanism, the VWFormer decoder, and empirical improvements over state-of-the-art methods. Extensive experiments validate the effectiveness of the proposed methods.", "questions": "1. How does VWA fundamentally differ from existing attention mechanisms in handling multi-scale representations?\n2. Include a breakdown of performance gains across different classes or segments within the datasets used.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Introduction of Varying Window Attention (VWA) to dynamically adjust receptive fields, addressing scale inadequacy and field inactivation.\n+ Development of new principles such as pre-scaling, densely overlapping patch embedding (DOPE), and copy-shift padding mode (CSP) to enhance efficiency in receptive field variation.\n+ Empirical improvements in mean Intersection over Union (mIoU) and reductions in computational cost (FLOPs).", "weaknesses": "- The paper may not sufficiently differentiate VWA from existing local window attention mechanisms.\n- Potential concerns about the scalability and generalizability of the proposed method to different architectures or datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z492W6aaHo", "reviewer": "Reviewer_u8Vk", "summary": "This paper observes the limitation of local window attention and proposes an efficient modification to enlarge the receptive field while keeping the computation efficiency. Specifically, it designs a pre-scaling strategy, which cleverly reduces the extra computation of enlarging window size. This paper presents detailed computation comparisons with existing methods and verifies the effectiveness of the proposed methods on several benchmarks.", "questions": "1. Are there any ablation studies indicating the effectiveness of copy-shift padding?\n\n2. From Table 7 in Supp, it seems avg_pool can be a substitution of PE, then I wonder what the performance will be like if the rescaling method \"DOPE --> avg_pool --> 1x1 conv\" is taken.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper starts from the visual observations, then goes into detailed computation analysis, and further proposes modifications to enhance. The logic between the whole paper is very smooth and easy to follow.\n\n2. The detailed complexity analysis substance the efficiency of the proposed method. Besides, the experimental results further verify computation efficiency compared to existing methods.\n\n3. Quantitative analysis verifies the effectiveness of the proposed methods, highlighting the significance of varying window sizes.", "weaknesses": "1. The proposed copy-shift padding is a bit weird. It mimics the rolling operation in numpy and torch. I wonder why adopting this instead of symmetric or constant padding.\n\n2. Fig 2b and Sec 3.2 are not well-aligned. In Fig 2b there's a PE operation, which doesn't occur in Sec 3.2, which can confuse the readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6ROtelCuKO", "reviewer": "Reviewer_Vryj", "summary": "This paper points out two problems existing in the previous multi-scale semantic segmentation representation: scale inadequacy and field inactivation. To solve these two problems, this paper presents its method - VWA (varying window attention). It begins by splitting local window attention (LWA) into query window and context window, which are variable so that queries learn representations on a specific scale. Then, to reduce memory consumption, the pre-scaling principle, densely overlapping patch embedding(DOPE), and copy-shift padding mode are proposed. Finally, VWFormer is devised for semantic segmentation, which shows its effectiveness in terms of performance and reducing cost.", "questions": "(1) For copy-shift padding mode(CSP), why not take the adjacent part to padding?\n(2) What is the short path branch?\n(3) What does pre. or post. in Table 7 refer to?  PE refers to padding with zero or CSP?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ This paper makes full use of multi-scale features for fusion and proposes VMA to obtain multi-scale representation.\n+ The experiments and visual results show the problems that existed in the previous work, namely, scale inadequacy and field inactivation, which is insight.\n+ The VMFormer frame is more clearly drawn and is easy to follow.", "weaknesses": "- Too much space is spent to explain the memory usage problem, resulting in some experimental results, such as Table 9 and some ablation experiments, which can only be placed in the appendix.\n- Lack of ablation experiments on the copy-shift padding mode. It is mentioned in the paper that filling with zero will cause attention collapse, but there are no experimental results to explain it.\n- There is no comparison between VWA and LWA.\n- The pre-scaling principle is to reduce memory usage, and the impact of pre-scaling on performance is not reflected.\n- The method part is not abundant, such as the short path branch. Overall, it seems that the original is not high, mainly VWA.\n- Inference time is not reflected in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3YVRoHucFO", "reviewer": "Reviewer_L266", "summary": "The paper delves into the topic of multi-scale representations and identifies two primary challenges: \"scale inadequacy\" and \"field inactivation.\" The authors introduce the LWA model and, based on this, design the VWFormer as a decoder to enhance multi-scale representation capabilities. Through the effective LWA module, VWFormer outperforms most compute-friendly multi-scale decoders, such as FPN and MLP decoders.", "questions": "1. Why not experiment with the same Encoder paired with different decoders (more than three)? This can help better show the consistent effectiveness of your approach. \n2. The computational load shown in Equation (2) appears highly reminiscent of the Swin Transformer. Could you possibly incorporate a comparison with the Swin Transformer?\n3. In Figure (3), you've employed an operation akin to that of Swin. However, after image concatenation in Swin, a mask is applied. As you concatenate features from different regions, how do you prevent interference or cross-effects between these diverse area features?\n4. SegFormer utilizes an MLP Decoder, which doesn't encompass an attention mechanism. Yet, in your VWA mechanism, you've incorporated an attention mechanism. How does this lead to a reduction in parameter count? This can be better analyzed.\n5. In the experimental section, could the results be organized based on the size of the modules in a systematic order?\n\nSincerely,", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Extensive experiments are conducted on Cityscapes, ADE20K, and COCOStuff-164K datasets. Despite reducing computational demands and the number of parameters, the proposed method demonstrates relative improvements compared to other approaches. \n\nThis work provides a thorough mathematical derivation of its method and showcases how to address the challenges of scale inadequacy and field inactivation.\n\nThis work presents a lot of ablation experiments to verify that the proposed methods outperform existing mechanisms.\n\nThe visualization results are interesting and provide many insights into multi-scale feature representations for semantic segmentation.", "weaknesses": "1. The manuscript requires structural refinement. Although the paper proposes a new methodology, the empirical validation predominantly resides in the appendix. The core content appears to be overly enmeshed in granular theoretical derivations, leading to a somewhat convoluted narrative structure.\n2. The delineation between the figures, tables, and the main text is ambiguous. It is advisable to render the captions in a font slightly smaller than the primary text to enhance clarity. The layout of the illustrations also warrants improvement. For instance, in Figure 2, it would be beneficial to incorporate descriptions of specific letters either within the caption or designated sections of the image. The term \"PE\" is presented here, yet a precise elucidation is deferred until Section 3.3.\n3. The visualization results do not substantiate the issues you've raised. The quantified outcomes displayed in Figure 5 merely illustrate the discriminative results of the proposed network in comparison to Segformer for a specific image region. If the method you presented exhibits a substantial improvement over Segformer, I believe it would be acceptable. However, given that the data only indicates approximately a 1% enhancement relative to Segformer. There is a lack of targeted visualization results addressing the issues of scale inadequacy and field inactivation.\n4. The author introduced \"varying window attention,\" but neglected a comparative analysis with other similar methods, such as the 'Shifted Window' in the Swin-Transformer.\n5. Some state-of-the-art segmentation methods like HRViT, SegViT-v2, ViT-adapter, Lawin Transformer, etc. could be compared in the experiments to better verify the superiority of your proposed architecture.\n\nSincerely,", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)