When Semantic Segmentation Meets Frequency Aliasing
Authors: Linwei Chen
Lin Gu
Ying Fu
Venue: ICLR 2024 poster (2024)
Keywords: Aliasing; Hard Pixel; Semantic Segmentation
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Despite recent advancements in semantic segmentation, where and what pixels are hard to segment remains largely unexplored.
Existing research only separates an image into easy and hard regions and empirically observes the latter are associated with object boundaries.
In this paper, we conduct a comprehensive analysis of hard pixel errors, categorizing them into three types: false responses, merging mistakes, and displacements. 
Our findings reveal a quantitative association between hard pixels and aliasing, 
which is distortion caused by the overlapping of frequency components in the Fourier domain during downsampling.
To identify the frequencies responsible for aliasing, we propose using the equivalent sampling rate to calculate the Nyquist frequency, which marks the threshold for aliasing. 
Then, we introduce the aliasing score as a metric to quantify the extent of aliasing.
While positively correlated with the proposed aliasing score, three types of hard pixels exhibit different patterns.
Here, we propose two novel de-aliasing filter (DAF) and frequency mixing (FreqMix) modules to alleviate aliasing degradation by accurately removing or adjusting frequencies higher than the Nyquist frequency.
The DAF precisely removes the frequencies responsible for aliasing before downsampling, 
while the FreqMix dynamically selects high-frequency components within the encoder block.
Experimental results demonstrate consistent improvements in semantic segmentation and low-light instance segmentation tasks.
The code is at: \url{https://github.com/Linwei-Chen/Seg-Aliasing}.

Published as a conference paper at ICLR 2024
WHEN SEMANTIC SEGMENTATION MEETS FREQUENCY ALIASING
Linwei Chen', Lin 2u & Ying Fu²
BBejjing Institute of Technology, Beijing, China
2RIKEN AIP, Tokyo, Japan
'The University of Tokyo. Tokyo, Japan
chenl i nwei @bit. edu.cn lin.gueriken, fuying@bit. www.bbbbt.duu..o
ABSTRACT
Despite recent advancements in semantic segmentation, where and what pixels are hard to segment remains largely unexplored. Existing research only sepa- rates an image into easy and hard regions and empirically observes the latter are associated with object boundaries. In this paper, we conduct comprehen- sive analysis hard pixel errors, categorizing them into three types: false re- sponses, merging mistakes, and displacements. Our findings reveal quantíta- tive association between hard pixels and aliasing, which is distortion caused by the overlapping of frequency components in the Fourier domain during down- sampling. To identify the frequencies responsible for aliasing, we propose us- ing the equivalent sampling rate calculate the Nyquist frequency, which marks the threshold for aliasing. Then, we introduce the aliasing score as a metric to quantify the extent of aliasing. While positively correlated with the proposed aliasing score, three types of hard pixels exhibit different patterns. Here, we pro- pose novel de-aliasing filter (DAF) and frequency mixing (FreqMix) mod- ules to alleviate aliasing degradation by accurately removing or adjusting fre- quencies higher than the Nyquist frequency. The DAF precisely removes the frequencies responsible for aliasing before downsampling, while the FreqMix dy- namically selects gigh-frequency components within the encoder block. Exper- imental results demonstrate consistent improvements in semantic segmentation and low-light instance segmentation tasks. The code is available at: https: //githus htwpL//wwwwihheeeeeeeeeeeeee
INTRODUCTION
Semantic segmentation is erucial task in computer vision, with numerous applications such as medical segmentation (Falk et al., 2019), autonomous driving (Hu et al., 2023), robotics (Milioto & Stachniss, 2019), and urban planning (Liu al., 2023; Chen al., 2021; 2022a; Fu al., 2022a). This dense prediction task heavily relies on high-frequency spatial information that encompasses fine details like textures, patterns, structures, and boundaries (Zhang al., 2019; Liu et al., 2021a; la; Zhu et al., 2021; Ding et al., 2019).
While recent models (Cheng et al., 2022; Wang et al., 2023) have achieved much progress, most of these existing methods treat all pixels equally and average each pixel's loss for the image-level one. However, it has long been observed that some pixels are harder to segment than others (Li et al., 2017: Gu et al., 2020). Previous works (Gu et al., 2020; Deng al., 2022) observe that pixels prone to error are associated with object boundaries, poor lighting conditions, etc. Considering this, (Li et al., 2017: Wang et al., 2020) separates image into two parts: easy and hard regions according to the confidence of predicted probability before applying different segmentation heads. Still, quantitative analysis of 'hard pixels' is open and challenging problem (Gu et al., 2020).
In this paper, we analyze hard pixels at boundaries and categorize them into three types: 1) false responses, 2) merging mistakes, and 3) displacements. As illustrated in Figure false responses occur when the model predicts boundaries in areas without the object of interest. Merging mistakes
"Corresponding Author

---

Published as a conference paper at ICLR 2024
(a) False response
(b) Merging mistake
(c) Displacement
Prediction
Ground truth
Prediction
Ground truth
Prediction
Ground truth
Figure I:
Illustration of three boundary error types.
refer to the failure to predict boundaries in regions containing the object of interest, resulting in the erroneous merging of two objects into one or missing predictions. Displacements involve predictions that are close to the correct location but deviate from the ground truth object boundary.
We further quantitatively associate three types of errors with a previously unexplored phenomenon: aliasing. Aliasing refers to the overlapping of frequency components in the Fourier domain when signal undersampled. According the Syquist-Shanne Nyquist, 1928), signals with frequencies higher than half of the sampling rate will be distorted. This a particularly critical issue in semantic segmentation networks, which typically employ a series of downsampling operations to expand the receptive field and reduce dimensionality (He et al., 2016; Liu et al., Ib; Wang et al., 2023; Cheng et al., 2022; Chen et al., 2022b).
Sampling Theorem (Shannon, 1949;
Measuring aliasing level needs to calculate the Nyquist frequency. Existing research (Grabinski et al., 2022) calculates Nyquist frequency based on the downsampling stride, egg. stride of assumes a sampling rate of 1, causing frequencies above (half of the sampling rate) to become aliased. This assumption holds for point-wise downsampling or downsampling without channel expansion, e.g., 1x1 convolution, max pooling. However, for larger kernels (e.g., 2x2) and wider output channels (e.g., 2 wider), the actual downsampling rate should be larger, i.e., X/2
In light of this, we propose to calculate the actual Nyquist frequency based on equivalent sampling rate, ie., equivalent Nyquist frequency. The determination of this equivalent Nyquist frequency depends on both the sampling kernel size and the size of the input-output feature maps. Then, measure the aliasing level with an aliasing score, which defined as the ratio of high-frequency power above our Nyquist frequency the total power of the spectrum.
As shown in Figure 9, our analysis reveals a positive correlation between the hard pixels at bound- aries and the aliasing score. All three types of errors increase as the aliasing score rises. These errors exhibit distinct characteristics when analyzed from the perspective of aliasing. Additionally, we also note that the importance of these errors varies across different scenarios. For example, displacement errors are primarily concentrated in areas with a high aliasing score, as depicted in Figure 9, In critical tasks such as robotic surgery or radiation therapy, even a displacement of only two or three pixels from vital organs like the brainstem or the main artery can have fatal consequences. In con- trast, false responses and merging mistakes, which hold greater significance autonomous driving scenarios, tend to be more prevalent in areas with relatively low aliasing scores.
The cellasnng-induced error can partially explain the success of low-pass filter-based methods (Zhang, 2019; Zou et al., 2020; Grabinski et al., 2022; Chen et al., 2023), they reduce specific high- frequency components to alleviate aliasing. But they either empirically set the low-pass kernel or underestimate the frequency threshold. Here, we propose two simple yet effective solutions: the de- aliasing filter (DAF) and the frequency mixing module (FreqMix). Based on our calculation of the equivalent Nyquist Frequency, DAF accurately removes the frequencies responsible for aliasing in the Fourier domain during specific downsampling (e.g., 3x3 convolution with stride of two). Given the importance of high-frequency components in representing crucial detailed information (Li al., 2020a; Bo et al., 2023), FreqMix can dynamically select and balance high- fegeurryyyency components in each encoder block (e.g., ResNet block). Our main contributions can be summarized as follows:
Going beyond the binary distinction of easy and hard regions, we categorize pixels chal- lenging for segmentation into three types: false responses, merging mistake, and displace- ments. These three eтror types hold varying levels of significance in different tasks.
We introduce the concept of equivalent sampling rate for the Nyquist frequency calculation and propose an aliasing score for quantitative measurement aliasing levels. Our metric effectively characterizes the three types of errors with distinct patterns.
2

---

Published as a conference paper at ICLR 2024
1.0
Narint haur
ena Parstng rirats
Naa ccmmatt
9:3
339
1.00
935
323
0.35
0.20
0..0
0.20
0.20
0.13
6.20
3.20
5.15
Pall EocAaIie
salie Doundary Crme Tyons
Figure 2: Left: The correlation curve between the cross-entropy enror of boundary pixels and the aliasing
score. Middle: Cross-entropy
curve for three types of boundary pixels, correlated with aliasing score. Right: Distribution of thrce types of hard pixels.
We design a simple de- de-aliasing filter to precisely remove aliasing as measured by our alias- ing score. Additionally, we propose a novel frequency-mixing module to dynamically se- lect and utilize both low and high-frequency information. These modules can be easily in- tegrated into off-the-shelf semantic segmentation architectures and effectively reduce three types of errors. Experiments demonstrate consistent improvements over state-of- the-art art methods in standard semantic segmentation tasks and low-light instance segmentation.
RELATED WORKS
Hard pixels. Early researches (Shrivastava et al., 2016; Lin et al., 2017b) have found significant variations in classifying samples within images for object detection. Online Hard Example Mining (OHEM) (Shrivastava et al., 2016) selects challenging samples with high losses during training and ignores casily classifiable samples. In contrast, Focal Loss (Lin et al., 2017b) adaptive assigns weights to challenging samples according to their losses. These strategies show effectiveness. Some semantic segmentation approaches (Li et al., 2017) also employ hard sample mining. For instance, Li er al. (Li al., 2017) use shallow or deep networks based on predicted probability scores for easy or hard regions, respectively. Another study (Wang et al., 2020) uses three segmentation heads for coarse segmentation. They process hard and easy regions based on predicted scores and merge the results for the final output. Additionally, methods like Online Hard Region Mining (OHRM) (Yin et al., 2019) and NigntLab et al., 2022) identify challenging regions based on loss values (Yin al., 2019) or a detection network (Deng et al., 2022). In this work, our finding reveals a positive correlation between the hard pixels and the aliasing score, which is previously unexplored.
Aliasing in Neuron Networks. The aliasing effect neuron networks is receiving increasing atten- tion. The very first attempt (Zhang, 2019) identified that it is the aliasing makes the neuron network sensitive the shift. (Zhang, 2019) also proposed the anti-aliasing filters on the convolution layers to enhance shift-invariance. (Zou et al., 2020) introduces learned blurring filters to further improve shift-invariance. Recently, (Hossain al., 2023) suggests combining a depth adaptive blurring filter with anti-aliasing activation function. Wavelets transformation (Li al., 2021) also offers lution by leveraging low-frequency components to reduce aliasing and enhance robustness against common image corruptions. Beyond image classification, anti-aliasing techniques have gained rel- evance the domain of image generation. In (Karras et al., 2021), blurring filters are utilized to remove aliases during image generation in generative adversarial networks (GANs), while (Durall et al., 2020) and (Jung & Keuper, 2021) employ additional loss terms in the frequency space to address aliasing issues. Unlike the empirical design of low-pass filters in existing works, we quanti- tatively calculate the frequency threshold for aliasing and precisely remove the corresponding high frequencies during downsampling.
F Frequency learning Rahaman et al. (Rahaman et al., 2019) find the spectral bias that neuron net- works prioritize learning the www-freuency modes. Xu et al. (Xu Zhou, 2021) conclude the deep frequency principle, the effective target function for deeper hidden layer biases towards lower fre- quency during the training (Xu & Zhou, 2021). Qin et al. (Qin et al., 2021) exploring utilizing more frequency for channel attention mechanism. Xu et al. (Xu et al., 2020) introduces learning-based frequency selection method into well-known neural networks, taking JPEG encoding coefficients as input. Huang al. (Huang et al., 2023) employs the conventional convolution theorem in DNN, demonstrating that adaptive frequency filters can efficiently serve as global token mixers. Compared to existing works, this work investigates the relationship between high frequency and segmentation
3

---

Published as a conference paper at ICLR 2024
O
Desrarrq
Upuupp
Dentiengle
Illustration of how a larger kernel and channel expansion increase the sampling rate. Left: Point- wise downsampling with stride 2 and no channel expansion results in sampling rate of g. causing aliased high frequencies to be misrepresented as low frequencies. Right: Downsampling using 2 x 2 identity kernels with 4x channel expansion actually increases the sampling rate to 1, instead of 2. all pixels are sampled.
Figure 3:
error for the first time with introduced aliasing score, demonstrating the possibility of improving segmentation accuracy by optimizing the frequency distribution of features.
ALIASING DEGRADATION AND SOLUTION
3.1 ALIASING MEASUREMENT
Irrespective of specific architecture, modern Deep Neural Networks (DNNs) fundamentally consist of a series of stacked convolutional or transformer encoder blocks and downsampling layers (He et al., 2016; Liu et al., 2021b; 2022). From the perspective of signal processing, the downsampling operations are prone to violate Nyquist Sampling Theorem (Shannon, 1949: Nyquist, 1928) since the frequency of features, especially those close to the boundary, often exceeds the Nyquist frequency. The resulting frequency overlaps, formally known as aliasing, occur when high frequencies above the Nyquist frequency are undersampled, leading to signal distortion or misinterpretation.
Nyquist frequency calculation. To calculate the Nyquist frequency during downsampling, existing research (Grabinski et al., 2022) considers the stride of the downsampling layer. Specifically, for a downsampling layer with a stride of 2, since reduces the spatial size by half, the sampling rate 2. According to the Nyquist Sampling Theorem (Shannon, 1949), the Nyquist frequency half of the sampling rate, i.e., 1. This estimation is accurate for 1x1 point-wise downsampling layers depth-wise downsampling layers without channel expansion, such as 1x1 convolutions, max/average pooling. However, it may underestimate the sampling rate when downsampling layers employ larger kernel sizes and output channel expansion, they increase the actual sampling rate, such 2x 2 patch merging and 3 3x3 convolution in (Liu et al., 2022; 2021b; Wang et al., 2023).
To illustrate this phenomenon, provide simple example in Figure 3. By learning four different identity kernels and outputting 4x wider output channels, even though the spatial size is 2x down- sampled, the spatial information can be losslessly preserved because all pixels are actually sampled, i.e,, the sampling rate is instead of 1.
Based on this observation, we consider the kernel size and feature size (channel, height, and width) instead of just the downsampling stride. We introduce an alternative yet straightforward solution, equivalent sampling rate (ESR), to calculate the actual sampling rate as:
V Crmit Cinn
Houl x Wowl
ESR min( /down
Hin x Win
where C. H, and W represent channel, height, and width sizes. 'in' and 'out' refer to input and output features. We assume equal sampling rates for height and width, calculated by square root,
i...,
Hon x Woot
Hax
Sinde which aligns with (Grabinski et al., 2022). min( K down
C C
reflects the
impact of downsampling kernel size Kdown and channel expansion. For example, with 2x2 kernel
(Krdown- 2), it can double the sampling rate if the channel expands by a factor of 4 ( C
4),
CT
accommodating the sampled information. However, with lower channel expansion (c.g., Ol 2),
C
it limits the sampling increment min v2) V2. Specifically, ResNet (He et al., 2016), Swin- Transformer (Liu et al., 2021b), and ConvNeXt (Liu et al., 2022) use a 3x3 residual block (can be regarded as 3x3 convolution (Ding et al., 2021)), 2x2 patch merging with 2x channel expansion.
Thus, the equivalent sampling rate is v2 and the corresponding Nyquist frequency is v2
Aliasing score. After calculating the Nyquist frequency, we can obtain the set of high frequencies
larger than the Nyquist frequency that lead to aliasing: H. = [(k.)) | [k] > 2
ESR or [i] > ESR ).
2
Thus, quantitatively measure the aliasing level by introducing aliasing score as metric,
4

---

Published as a conference paper at ICLR 2024
which we define as the ratio of frequency power above the Nyquist frequency to the total power in the frequency spectrum.
Specifically, we first transform the feature map f RC* into the frequency domain using the Discrete Fourier Transform (DFT), it can be represented as:
#-1 1-IW-1
F(c, k,1) H W
Σ Σ ==0
flc, h, (ehh,w)e-njskhhtte
(2)
h=0
where F RCXHXW represents the output array of complex numbers from the DFT. H and W denote its height and width. And the h, w indicates the coordinates of feature map f. The frequencies in the height and width dimensions are given by | and [1), with i taking values from
the set {0. H:... :....: H-1} } and from {0, W W. Consequently, the aliasing score can be formulated as follows:
EmNDER [F(c,
Aliasing Score
(3)
Σ El
3.2 BOUNDARY ERROR TYPE
Although is widely known that predicting object boundaries accurately is challenging and often results in high errors, there is no existing work that investigates boundary error types in-depth. Here, further divide the boundary eπror into three types, false response, missing edges, and displace- ment, as shown in Figure 1. Through aliasing analysis, we find they show different characteristics.
False response refers to instances where the predicted edges erroneously appear in non-boundary areas, also known as false positives. These false responses are often associated with texture and noise-related issues as reported in previous studies (Lopez-Molina al., 2013). Notably, as shown in Figure 9, these errors tend to occur in regions characterized by a relatively low aliasing score.
Merging mistake occurs when the predicted results fail to capture the corresponding edges, result- ing in false negatives. They are typically linked to low-contrast regions (Lopez-Molina et al., 2013).
As depicted in Figure 9, these errors also manifest in areas with a relatively low aliasing score.
Displacement error arises when the predicted edges deviate from their true positions. Such dis- placements may occur when the image features are misaligned (Li et al., 2020b; Huang al., 2021). As depicted in Figure 9, these errors tend to manifest in areas with relatively high aliasing score.
Metries for three errors. Additionally, we have designed series of metrics as tools to assess and analyze the error rate of three distinct types of hard pixel errors at boundaries: false response (FErr), merging mistake (MErr), and displacement (DErr): |a (Pa GPan
Fßrr Pa (Ga n (a)l MErr
I(Pa n P) (Gan
DEr
(4)
Paa
[Ga]
Pan
where P and G refer to the binary mask of prediction and ground truth, Pa and Ga represent the pix- els located in the boundary region of the binary mask with a pixel width of d. As per the guidelines presented in (Cheng al., 2021), we set the pixel width of the boundary region 15 pixels.
Impact of blur filters. Previous work (Zhang, 2019) has explored the use of a simple Gaus- sian blur to mitigate aliasing in DNNs. How- ever, due to the absence of metric for eval- uating the level of aliasing and the oversight of three distinct types of hard pixel errors at boundaries, the reason why the counterintuitive phenomenon of a simple blur filter improving performance remains unexplained. Here, armed with the tools of aliasing scores and metrics for three distinct types of hard pixel errors, take step further. As shown in Table 1, inserting a 3x3 Gaussian filter before downsampling effectively re- duces the average aliasing score from 9.4% to 0.27%, thus improving both the overall segmentation mloU and boundary segmentation BloU.
Table 1: Analysis for the impact of blur filters with UPerNet-RSO on Cityscapes validation set. The 1/4 in- dicates that higher/lower values better.
Kemel
Rlur
mlot?
DIGU
Beundary
BAZe
F[m:
Three type cmors
Min.
Dim
Alineing Score
781
61.3
74.4
21.2
23.1
26.9
9.4%
62.3
3K.5
78.7
622
0.16%
7
61.7
0148
Further increasing the blur kernel size to 5x5 and 7x7 does not yield additional improvements; instead, leads to degradation compared to the 3x3 kernel. This occurs because the larger ker- nel sizes blur boundary details, resulting in higher error rates for false responses (FErr, merging
5

---

Published as a conference paper at ICLR 2024
Input image Figure 4:
Feature at stage
Feature stage 2
Feature at stage 3
Feature at stage
Illustration of the impact of noise the features at different stages.
mistakes (MErr), and displacement errors (DErr). In summary, an appropriately sized blur kernel improves the model by reducing hard pixel errors. However, larger blur kernels can result in a higher error rate across three types of errors, potentially counteracting the improvements
Impact of noise. Previous work (Chen et al., 2023) has revealed the negative impact of noise in low-light scenarios. Here, we further analyze the relationship between aliasing and noise. Modern backbone models typically organize the convolutional/trassfonnee encoder block into four stages. The first stage is downsampled by factor of compared the input images, and there are three 2x downsampling operations from stage-1 1 to stage-4. Consequently, we present the aliasing score results for the three 2x downsampling operations at stages to
As shown in Table 2, we introduce Gaussian noise images and observe a substantial increase in the aliasing ratio of the feature map at the first stage. This observation indicates that noise amplifies the severity of alias- 20
Table 2: Analysis for the impact noise with UPerNet-R50 on
Cityscapes validation set.
Houmary
type crmms
Aliasing
lexd
BlaUt
Fen.
MEn.
Dem
Siage-1
Stggs:
Stages
78.1
374
730
25.4
83.
27.6
5.4%
11.5%
11.3%
112
332
661
36.
3538
333
6.1%
11.0%
10.3%
10
54.1
39.7
50.6
47.3
49.0
45.7
6.4%
10.3%
9.9%
20.6
13.0
22.0
72.7
76.7
71.
9.69
8.456
6.4%
ing effects. However, for deeper features at the second and third stages, the aliasing ratio decreases.
This decrease does not signify a reduction in aliasing; rather, it occurs because the earlier high level of aliasing results in severe degradation. Consequently, the features become hard pixels, meaning that the following stage struggles to extract useful information and loses response to objects. Thus feature maps appear plain and lack useful high frequency, shown in Figure As a result, there is a significant deterioration in all three types of errors, particularly in the case of displacement error.
3.3 ALLASING-AWARE SOLUTION
In this section, we introduce two solutions: De-Aliasing Filter (DAF) for removing aliasing during downsampling and Frequency Mixing (FreqMix) for adjusting frequency within encoder block.
De-aliasing filter. Several prior techniques, as introduced in (Zhang, 2019; Zou et al., 2020; Hossain et al., 2023), employ methods to atten- uate high-frequency components within feature maps prior to downsampling, aiming to prevent aliasing artifacts. These approaches typically employ traditional spatial domain blurring op- erations for this purpose. Recent work (Gra- binski et al., 2022) also explores the removal of high frequencies in the Fourier domain. However, it determines the low-pass cutoff frequency solely based on the downsampling stride, leading to an underestimation of the sampling rate. This results an excessive removal of high frequencies that are crucial for predicting segmentation details.
Down
Figure 5: Illustration of de-aliasing filter (DAF). FFT: Fast Fourier Transform, iFFT: Inverse FFT.
On the basis of our Nyquist frequency calculation in Section 1.1, we propose the De-Aliasing Filter (DAF). first transforms the feature map to downsampled into the Fourier domain and then sets the power of high frequencies that lead to aliasing to zero. Leveraging the equivalent sampling rate equation 7, we can calculate the actual Nyquist frequency, which serves as the low-pass cut-off' frequency for the DAF. It accurately removes the frequency power causing aliasing. This precise removal of frequency power effectively eliminates aliasing. Figure 5 illustrates its procedure in detail. In the spatial domain, this operation would involve convolving the feature map with an infinitely large non- hondaandlieited filter, which cannot be implemented in practice (Grabinski et al., 2022). Note that DAF is parameter-free and easily integrable module.

---

Published as a conference paper at ICLR 2024
Frequency Mixing Module. The degradation introduced by aliasing underscores the importance of frequency balancing in segmentation. On one hand, high frequencies are crucial for representing important detailed information, such object boundaries (Li et al., 2020a; Bo al., 2023). On the other hand, high frequencies above the Nyquist frequency can lead to harmful aliasing. Motivated by this observation, we shift our focus to the feature extraction blocks and introduce frequency- mixing module. We insert this module after the convolutional layer, where it adaptively weights the frequency power both below and above the Nyquist frequency. Its formal representation is:
f' (,,,,,w) w) (c, w) f- h, w) + (e, w) (c, h, w),
(5)
where c, h, and are indices in the channel, height, and width dimen- sions, and ʃ and I" are input and out- put feature maps. ft and ff are fre- quency components below and above the Nyquist frequency, which can be simply obtained by low-pass/high- pass filtering in the Fourier domain, Figure 6: Illustration of Frequency Mixing (FreqMix) module. as shown in Figure 6. At and AT are the corresponding weighting values for frequency mixing. However, directly predicting At, At E RC can be computationally heavy. Thus, we decom- pose the weighting values prediction into channel-wise and spatial-wise components
ITTT
(6) where the channel-wise prediction can be simply implemented by global pooling followed by a fully connected layer, and spatial-wist prediction can be implemented by a convolution layer. We use the sigmoid function to regularize the weighting values to be in the range [0, 1]. FreqMix can easily integrate into the encoder block of existing models (He et al., 2016; Wang et al., 2023).
(c, h, w) Ai(c) A2 w) (c, w) + Al(c) A3, w) (c,hh,
4
EXPERIMENTS
4.1 METRIC AND DATASET
Metric. Following previous works (Long et al., 2015; Cheng et al., 2022), we employ standard metrics like mean intersection-over-uiii
(mloU). For finer evaluation of boundary segmentation accuracy, inspired by (Cheng et al., 2021), we use mean boundary IoU and accuracy (BloU, BAcc). Additionally, we introduce FErr, МEr,, and DErr from equation 4 to assess distinct hard pixel errors at boundaries: false responses, merging errors, and displacements.
Datasets. For semantic segmentation,
we employ widely-used and challeng- ing Cityscapes (Cordts et al., 2016),
Table 3: Ablation study for low-pass cut-off frequency on the
Cityscapes validation set. A higher low-pass cut-off frequency
allows more higu-freyuency components pass through.
PASCAL VOC (Everingham et al., 2010), and ADE20K (Zhou et al., 2017) datasets. Cityscapes (Cordts et al., 2016) consists of 5,000 finely anno- tated images, each with dimensions of 2048 1024 pixels. This dataset is meticulously divided into 2,975 images for the training set, 500 for the valida- tion set, and 1.525 for the testing set. It
Cut-off froquency
Boundary
Three
Alissing Score
mloUt
BloU
BAce
FEm..
MErr.]
DEm.
78.1
61.8
74.4
27.2
25.1
26.9
9.4
i x 1.0
78.6
62.7
74.9
26.7
24.8
26.7
0.0
78.7
62.3
75.4
26.7
24.8
0.0
1.2
78.8
62.4
75.4
26.3
24.3
26.2
0.0
1.3
79.2
62.4
75.4
26.2
24.3
26.
0.0
79.3
62.6
75.7
26.0
24.0
25.9
0.0
1.5
78.9
62.4
75.0
26.6
24.6
26.4
1.51
1.6
79.1
62.1
75.2
26.7
24.5
26.5
3.17
encompasses 19 distinct semantic categories. Pascal VOC (Everingham et al., 2010) involves 20
foreground classes and background class. After augmentation, it has 10, 582/1, 449/1, 456 images for training, validation and testing, respectively. ADE20K (Zhou et al., 2017) is notably challeng- ing dataset that encompasses whopping 150 semantie classes. This extensive dataset features distribution of 20,210 images for training, 2,000 for validation, and 3,352 for the test set. Consider- ing the importance of low-light applications (Zhang al., 2021; 2023; Fu et al., 2022c), which are often characterized by significant noise (Fu et al., 2022b; Wei et al., 2020; 2021; Zou & Fu, 2022; Zou et al., 2023), we also use the low-light instance segmentation dataset LIS (Hong et al., 2021; Chen et al., 2023) to evaluate the proposed method. It provides 8 common classes and consists of 2,230 pairs of low-light and normal-ligh images, with 70% for training and 30% for testing.
7

---

Published as a conference paper at ICLR 2024
Figure 7:
Visualized results on ADE20K. The top and middle rows represent predictions without and with the proposed methods, respectively. The bottom row represents the ground truth.
4.2 IMPLEMENT DETAILS
All UPerNet (Xiao et al., 2018) is trained on the Cityscapes dataset using a crop size of 768 x 768, a batch size of 8, and total of 80K iterations. We employ stochastic gradient descent (SGD) with a momentum of 0.9 and a weight decay of 5e-4. The initial learning rate set at 0.01. During training, adjust the learning rate using the common 'poly' learning rate policy, which reduces iter )u.9. We apply standard data augmentation tech- the initial learning rate by multiplying (1 max iier
niques, including random horizontal flipping and random resizing within the range of 0.5 to For InternImage (Wang et al., 2023) on ADE20K, we use the same training settings in the paper (Wang al., 2023). For PointRend (Kirillov al., 2020) and Mask2Former (Cheng al., 2022) on the LIS dataset, we adopt the same training settings as described in the paper (Chen et al., 2023).
4.3 ABLATION STUDY
Here, we conduct experi- ments to verify the effec-
Table 4: Comparison with blur filters Cityscapes validation
Mathod
m1al1
Btsth
BAcct
FEri
MEel
DEm]
4FLOP%
PParms
TPPo-Nos-R R50 (aan 2018)
78.1
61.8
744
272
13.1
207 10T3RK
31.6M
tiveness of the proposed
Blu 12hang, 2019)
78.8
750
25.5
30.
De-aliasing filter(DAF)
Addlllur (Zou 2021)
(Grahinii
75.1
26.4
31
and frequency mixing module (FreqMix). We
Chrs mans
753
259
Ours USF Poaglis wirchandoooo
79.4
62.
758
25.5
2538
25.5
298.54G
31.22M
Ours mar Pooghte)
79.7
63.2
76.2
25.2
23.3
24.8
296.770
31.54M
adopt widely used UPerNet (Xiao et al., 2018) with ResNet-50 (He et al., 2016) as the segmentation model owing its effectiveness and simplicity.
Low-pass cut-off frequency of DAF. Previous work (Grabinski al., 2022) determines the low-pass cut-off fre- quency simply by the size of the down- sampling layer, i..., 1. As shown in Table 3. By reducing all frequencies above 1/ to prevent alias- ing, it improves the vanilla UPerNet by 0.5 mIoU and reduces displacement errors, false responses, and merging mistakes by 0.5, 0.3, and 0.2, respectively. By calculating the actual sampling rate accurate equivalent Nyquist frequency of 2/2 which achieves
Table 5: Results on PASCAL VOC dataset.
Method
mlaUt
BloUf
Boundary
BAwe
FEml
Three type errors
MEn..
DErr.
27.8
Oues (+ BAF- ProgMis)
76.1
62.7
74.9
26.0
24.0
23.6
with equation 7, estimate the best overall results (mloU +1.2, BIoU +0.8) among settings where the low-pass cut-off frequency ranged from 1 to 1 1.6.
Effectiveness of FreqMix. As shown in Table 4, with spatial-wise weighting, it improves the overall performance. Further weighting the frequency for the
Table 6: Results on the ADE20K dataset.
Method
mloUt
HloUt
Boundary
RAaet
Three type errons
Mirr.
DErr
UPeeNNe
3899
29.3
20.5
604
58.9
60.2
Quis DAP RegMitd
40.4
31.2
433
57.9
5546
56.7
channel-wise component leads to an additional improvement, totaling +0.4 mloU while reducing all three types of errors. Mcanwhile, it only adds a minimal increase in FLOPs and parameters.
4.4 SEMANTICE SEGMENTATION
Comparison with state-of-the-ard blur filters. As shown in Table 4, we compare the proposed DAF with former ofate-of-thh-aa blur filters designed to alleviate aliasing, including Blur (Zhang, 2019), AdaBlur (Zou al., 2020), and FLC (Grabinski et al., 2022). Due to more precise cal- culation of the actual sampling rate and the effective removal of the frequency power that leads to aliasing, the proposed DAF achieves the best results. Furthermore, when combined with FreqMix,
8

---

Published as a conference paper at ICLR 2024
Table 7: Quantitative comparisons for low-light instance segmentation are conducted the LIS test set (Chen et al., 2023), utilizing PointRend (Kirillov et al., 2020) and Mask2Former (Cheng et al., 2022) as instance segmentation models, with ResNct-50-FPN (He et al., 2016; Lin et al., 2017a) as the backbone. The training and testing settings align with those outlined in (Chen et al., 2023).
Pipeline
Preprncessing method
Method
Apmesk
AFleo
Direct
PointRend
20.6
23.5
EnlightesGAN (Jang et al., 2021)= + SGN (Ou et al., 2019)
Pointend
26.9
31.2
Enllncce Denoise
ZeroDDE (Guo 2020) SGN (Gu 2019)
PoimtRend
27.7
31,9
Inlograted
SID (Chen al., 2018) Mitra, 2021)
Foinieen
28.3
31.6
Enhance Denoise
PoimiRend
27.7
End-to-end (C Chen et al,, 2023)
Prinikend
32.8
37.1
End- to end (Ours)
PointRend
(41.2)
38.2 (+1.1)
Direct
Mask2Former
21.4
22.9
EnlightesiAAN (Jiang al.,., 2021) SGN (Gi 2019) Enhance Denoise
Mark2
3039
DCroDDE (Gino et al., 2020) SON (Gn al,, 2019)
Mask2Former
29.3
31.9
Inlegratied
SID (Chen et al., 2018)
Fask2Fommrr
31.7
33.2
Enhance Denoise
REDI (Lamba Mitra, 2021)
Mask2Former
26.7
28.1
Ead-sp-endi hen al., 20223)
Mare2armmrr
33.0
37.8
Fid--to-end (Ours)
Maskerrrmer
36.7
(8.8
leads to further overall improvements, outperforning the best competitor, AdaBlur, by 0.8 mloU, and achieving the lowest error rates for all three types of hard pixels.
Comparison across different challenging datasets. As shown in Tables and 6, we present com- parison of segmentation results across two challenging datasets: PASCAL VOC and ADE20K. They display our proposed method, denoted as 'Ours (+ DAF + FreqMixx), outperforming the UPerNet baseline across all metries. Particularly, BloU, and BAcc (increases 1.8 mloU, 1.7 BloU, and 1.5 BAce on PASCAL VOC, and 1.5 mIoU, 1.7 BIoU, and 2.8 BAcc on ADE20K). Additionally, the three types of errors notably reduced (decreases of 1.8 FErr, 0.9 MErr, and 0.9 DEr on PASCAL VOC, and 2.5 FErr, 2.3 MErr, and 3.5 DErr ADE20K), demonstrating the effectiveness of our approach in addressing hard pixels.
method achieves substantial improvement in mIoU,
Visualized results. In Figure 7, our proposed method considerably enhances semantic segmentation quality, resulting in substantial reduction in all three error types, including false responses, merging errors, and displacements. These are consistent with quantitative results.
4.5 LOW-LIGHT INSTANCE SEGMENTATION
We assess the effectiveness of our proposed method for the challenging task of low-light instance segmentation. The presence of noise in low-light images results in high aliasing level, which can severely impair performance. As demonstrated in Table 7, our proposed method exhibits a substantial performance improvement (+1.2 AP for PointRend, +1.1 AP for Mask2Former) when compared to the previously established state-of-the-ar pipelines described in (Chen et al., 2023).
CONCLUSION
In this study, we analyze three types of hard pixels that occur at object boundaries and establish a quantitative link between them and aliasing degradation. Specifically, we propose a method to calculate equivalent sampling rates for estimating the Nyquist frequency and aliasing levels during specific downsampling operations, employing kernel larger than the stride and involving channel expansion. Moreover, demonstrate the existence of positive correlation between pixel-wise segmentation errors and the newly introduced aliasing score. Remarkably, these three types of challenging pixels exhibit distinct and unique patterns within the aliasing score, hinting at potential solutions tailored to scenarios that are sensitive to different types of errors.
These analyses empower to precisely adjust the aliasing frequencies above the Nyquist frequency for specific downsampling operations. In this regard, we present two concrete solutions. The first solution, the De-Aliasing Filter (DAF), focuses on the modification of the downsampling layer, precisely removing frequencies that lead to aliasing. The second solution, the Frequency Mixing (FreqMix) module, involves holistic adaptation of the encoder block, aiming to adjust the fre- quency distribution during feature extraction. Experimental results effectively validate the improved performance in both semantic segmentation and low-light instance segmentation tasks.
The insights gained from this analysis and the measurement tools we have introduced provide fresh perspective and hold the potential for unlocking numerous research opportunities in this field.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (62088101 and 62171038), and JST Moonshot R&D Grant Number JPMJMS2011, Japan.
REFERENCES
Dong Bo, Wang Pichao, and Fan Wang, Afformer: Head-free lightweight semantic segmentation with linear transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, pp- 1-9, 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of IEEE International Conference on Computer Vision, pp. 9650-9660, 2021.
Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. Learning to see in the dark. In Proceedings of IEEE Conference on Computer Vision and Puttern Recognition, pp. 3291-3300, 2018.
Liang-Chit Chich Chen, George Papandreou. Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint wwXiv:1706.055500 2017.
Linwei Chen, Ying Fu, Shaodi You, and Hongzhe Liu. Efficient hybrid supervision for instance segmentation in aerial images. Remote Sensing, 13(2):252, 2021.
Linwei Chen, Zheng Fang, and Ying Fu. Consistency- map generation at multiple zoom levels using aerial image. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 15:5953-5966, 2022a.
Linwei Chen, Ying Fu, Shaodi You, and Hongzhe Liu. Hybrid supervised instance segmentation by learning label noise suppression. Neurocomputing, 496:131-146, 2022b.
Linwei Chen, Ying Fu, Kaixuan Wei, Dezhi Zheng, and Felix Heide. Instance segmentation in the dark. International Journal of Computer Vision, 131(8):2198. 2218, 2023.
Bowen Cheng, Ross Girshick. Piotr Dollár, Alexander Berg, and Alexander Kirillov. Boundary iou: Improving object-centric image segmentation evaluation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 15334-15344 2021.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1290-1299, 2022.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler. Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings IEEE Conference Computer Vision and Pattern Recognition, pp. 2213-3223, 2016.
Xueqing Deng, Peng Wang, Xiaochen Lian, and Shawn Newsam. Nightlab: A dual-level architec- ture with hardness detection for segmentation night. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 16938-16948, 2022.
Henghui Ding, Xudong Jiang, Ai Qun Liu, Nadia Magnenat Thalmann, and Gang Wang. Boundary- aware feature propagation for scene segmentation. In Proceedings of IEEE International Confer- on Computer Vision, pp. 6819-6829, 2019.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma. Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vtg-style convnets great again. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 13733-13742, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom,, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of International Conference Learning Representations, pp. 1-12, 2020.
10

---

Published as a conference paper at ICLR 2024
Ricard Durall, Margret Keuper, and Janis Keuper. Watch your up-convolution: Can based generative deep neural networks are failing to reproduce spectral distributions. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition, pp- 7890-7899, 2020.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voe) challenge. International Journal of Computer Vision, 88 (2):003-338, 2010.
Thorsten Falk, Dominic Mai, Robert Bensch, Özgün Çięek, Ahmed Abdulkadir, Yassine Marrakchi. Anton Böhm, Jan Deubner, Zoe Jäckel, Katharina Seiwald, al. U-net: deep learning for cell counting, detection, and morphometry. Nature methods, 16(1):67-70, 2019.
Ying Fu, Zheng Fang, Linwei Chen, Tao Song, and Defu Lin. Level-aware consistent multilevel map translation from satellite imagery. IEEE Transactions on Geoscience and Remote Sensing, 61:1-14, 2022a.
Ying Fu, Yang Hong, Linwei Chen, and Shaodi You. Le-gan: Unsupervised low-light image en- hancement network using attention module and identity invariant loss. Shooledgg--Bseed Systems, 240: 108010, 2022b.
Ying Fu, Zichun Wang, Tao Zhang, and Jun Zhang. Low-light raw video denoising with a high- quality realistic motion dataset. IEEE Transactions on Mullimedia, 2022c.
Julia Grabinski, Steffen Jung, Janis Keuper, and Margret Keuper. Frequencyloweut pooling-plut and play against catastrophic overfitting. In Proceedings of European Conference on Computer Vision, pp. 36-57, 2022.
Shuhang Gu, Yawei Li, Luc Van Gool, and Radu Timofte. Self-guided network for fast image denoising. In Proceedings of IEEE International Conference on Computer Vision, pp. 2511- 2520, 2019.
Zhangxuan Gu, Li Niu, Haohua Zhao, and Liqing Zhang. Hard pixel mining for depth privileged semantic segmentation. IEEE Transactions on Multimedia, 13:3738-37511 2020.
Chunle Guo, Chongyi Li, Jichang Guo, Chen Change Loy, Junhui Hou, Sam Kwong, and Runmin Cong. Zero-reference deep curve estimation for low-light image enhancement. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition, pp. 1780-1789, 2020.
Ali Hassani and Humphrey Shi. Dilated neighborhood attention transformer. 2022. URL https: //arxiv. wwto wtw.aa/////..00000
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. I Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 16000-16009. 2022.
Yang Hong, Kaixuan Wei, Linwei Chen, and Ying Fu. Crafting object detection in very low light. In Proceedings of the British Machine Vision Conference, volume 1, pp. 1-15, 2021.
Md Tahmid Hossain, Shyh Wei Teng, Guojun Lu, Mohammad Arifur Rahman, and Ferdous Sohel. Anti-aliasing deep image classifiers using novel depth adaptive blurring and activation function. Neurocomputing, 536:164-174, 2023.
Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, al. Hanning-orientee autonomous driving. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 17853-17862. 2023.
Shihua Huang, Zhichao Lu, Ran Cheng, and Cheng He. Fapn: Feature-aligned pyramid network for dense image prediction. Proceedings of IEEE International Conference on Computer Vision, pp. 2021.
11

---

Published as a conference paper at ICLR 2024
Zhipeng Huang. Zhizheng Zhang, Cuiling Lan, Zheng-Jun Zha, Yan Lu, and Baining Guo. Adaptive frequency filters as efficient global token mixers. In Proceedings of IEEE International Confer- ence on Computer Vision, pp. 1-11, 2023.
Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang. Enlightengan: Deep light enhancement without paired supervision. IEEE Transactions on Image Processing, 30:2340-2349, 2021.
Steffen Jung and Margret Keuper. Spectral distribution aware image generation. In Association for the Advancement of Artificial Intelligence, volume 35, pp. 1734-1742, 2021.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. NewrlPS, 34:852-863, 2021.
Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. In Proceedings IFFE Conference on Computer Vision and Pattern Recognition, pp- 9799-9988, 2020.
Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected erfs with gaussian edge potentials. NeurIPS, 24:1-14, 2011.
Mohit Lamba and Kaushik Mitra. Restoring extremely dark images in real time. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 3487-3497, 2021.
Qiufu Li, Linlin Shen, Sheng Guo, and Zhihui Lai. Wavecnet: Wavelet integrated cnns to suppress aliasing effect for noise-robust image classification. IEEE Transaction on Image Process., 30: 7074-7089, 2021.
Xiangtai Li, Xia Li, Li Zhang, Guangliang Cheng. Jianping Shi, Zhouchen Lin, Shaohua Tan, and Yunhai Tong. Improving semantic segmentation via decoupled body and edge supervision. In Proceedings of European Conference on Computer Vision, pp. 435 452. Springer, 2020a.
Xiangtai Li, Ansheng You, Zhen Zhu, Houlong Zhao, Maoke Yang, Kuiyuan Yang, Shaohua Tan, and Yunhai Tong. Semantic flow for fast and accurate scene parsing. In Proceedings of European Conference on Computer Vision, pp. 775-793. Springer, 2020b.
Xiaoxiao Li, Ziwei Liu, Ping Luo, Chen Change Loy, and Xiaoou Tang. Not pixels are equal: Difficulty-awan semantic segmentation via deep layer cascade. In Proceedings of IEEE Confer- ence on Computer Vision and Pattern Recognition, pp. 3193-3202. 2017.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2117-2125, 2017a.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, pp. 2980-2988, 2017b.
Jiawei Liu, Jing Zhang, Yicong Hong, and Nick Barnes. Learning structure-aware semantic seg- mentation with image-level supervision. In International Joint Conference on Neural Networks, pp. 1-8, 2021a.
Songlin Liu, Linwei Chen, Li Zhang, Jun Hu, and Ying Fu. A large-scale climate-aware satellite image dataset for domain adaptive land-cover semantic segmentation. ISPRS Journal of Pho- togrammetry and Remote Sensing, 205:98-1 10---111 11,, 2023.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer. Hierarchical vision transformer using shifted windows. In Proceedings of IEEE International Conference on Computer Vision, pp. 10012-10022, 2021b.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer. Trevor Darrell, and Saining Xie. A convnet for the 2020s. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1 19966-1 1986, 2022.
12

---

Published as a conference paper at ICLR 2024
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of IEEE International Conference on Computer Vision, pp. 3431- 3440, 2015.
Carlos Lopez-Molina, Bernard De Baets, and Humberto Bustince. Quantitative error measures for edge detection. Puttern recognition, 46(4): 111p)::1222111111 125-1139, 2013.
Davide Mazzini. Guided upsampling network for real-time semantic segmentation. In Proceedings of the Brilish Machine Vision Conference, pp. 1-12, 2018.
Andres Milioto and Cyrill Stachniss. Bonnet: An open-source training and deployment framework for semantic segmentation robotics using cnns. In IEEE International Conference on Robotics and Automation, pp. 7094-7100, IEEE, 2019.
Harry Nyquist. Certain topics in telegraph transmission theory. Transactions of the American Insti- tute of Electrical Engineers, 47(2):617-644, 1928.
Shengju Qian, Hao Shao, Yi Zhu, Mu Li, and Jiaya Jia. Blending anti-aliasing into vision trans- former. Proceedings of Advances in Neural Information Processing Systems, 34:5416-5429, 2021.
Zequn Qin, Pengyi Zhang, Fei Wu, and Xi Li. Feanet: Frequency channel attention networks. In Proceedings of IEEE International Conference on Computer Vision, pp. 783-792, 2021.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Proceedings of Inter- national Conference on Machine Learning, pp. 5301-5310, 2019.
Yongming Rao, Wenliang Zhao, Yansong Tang, Jie Zhou, Ser Nam Lim, and Jiwen Lu. Hornet: Ef- ficient high-order spatial interactions with recursive gated convolutions. Proceedings of Advances in Neural Information Processing Systems, 35:10333-1033 2022.
Claude E Shannon. Communication in the presence of noise. Proceedings of the IRE, 37(1):10-2,, 1949.
Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region- object detectors with online hard example mining. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition, pp. 761-769, 2016.
Dongyi Wang, Ayman Haytham, Jessica Pottenburgh, Osamah Saeedi, and Yang Tao. Hard at- tention net for automatic retinal vessel segmentation. IEEE Journal of Biomedical and Health Informatics, 24(12):3384 3396, 2020.
Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Interninage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 144099 2023.
Kaixuan Wei, Ying Fu, Jiaolong Yang, and Hua Huang. A physics-based noise formation model for extreme low-light raw denoising. In Proceedings of IEEE Conference on Computer Vision and Patlern Recognition, pp. 2758-2766 2267, 2020.
Kaixuan Wei, Ying Fu, Yingiang Zheng, and Jiaolong Yang. Physics-based noise modeling for extreme low-light photography. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):8520-8537, 2021.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding, In Proceedings of European Conference on Computer Vision, pp. 418-434, 2018.
Kai Xu. Minghai Qin, Fei Sun, Yuhao Wang, Yen-Kuang Chen, and Fengbo Ren. Learning in the frequency domain. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1740-1749, 2020. 2020.
13

---

Published as a conference paper at ICLR 2024
Zhiqin John Xu and Hanxu Zhou. Deep frequeney principle towards understanding why deeper learning is faster. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10541-10550, 2021.
Jin Yin, Pengfei Xia, and Jingsong He. Online hard region mining for semantic segmentation. Neural Processing Letters, 50:2665-2679, 2019.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang. Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of IEEE International Conference on Computer Vision, pp. 558-567, 2021.
Yuhui Yuan, Jingyi Xie, Xilin Chen, and Jingdong Wang. Segfix: Model-agnostic boundary refine- ment for segmentation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16, pp. 489-506. Springer, 2020.
Fan Zhang, Shaodi You, Yu Li, and Ying Fu. Learning rain location prior for nighttime deraining. In Proceedings of IEEE International Conference on Computer Vision, pp. 13148-13157, 2023.
Richard Zhang. Making convolutional networks shift-invariant again. In Proceedings of Interna- tional Conference on Machine Learning, pp. 7324-7334, 2019.
Xiaosong Zhang, Fang Wan, Chang Liu, Xiangyang Ji, and Qixiang Ye. Learning to match anchors for visual object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Zhenyu Zhang, Zhen Cui, Chunyan Xu, Yan Yan, Nicu Sebe, and Jian Yang. Patter--ffiiitivv propagation across depth, surface normal and semantic segmentation. In Proceedings of IEEE Conference on Computer Vision and Puttern Recognition, pp. 4106-4115, 2019.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition, pp. 633-641, 2017.
Lanyun Zhu, Deyi Ji, Shiping Zhu, Weihao Gan, Wei Wu, and Junjie Yan. Learning statistical texture for semantic segmentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 12537-12546, 2021.
Xueyan Zou, Fanyi Xiao, Zhiding Yu, and Yong Jae Lee. Delving deeper into anti-aliasing in convnets. In Proceedings of the British Machine Vision Conference, pp. 1-13, 2020.
Yunhao Zou and Ying Fu. Estimating fine-graines mne-grained noise model via contrastive learning. In Pro- ceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 12682-12691, 2022.
Yunhao Zou, Chenggang Yan, and Ying Fu. Iterative denoiser and noise estimator for self-supervised image denoising. In Proceedings of IEEE International Conference on Computer Vision, pp. 15265-13274, 2023.
14

---

Published as a conference paper at ICLR 2024
APPENDIX
This supplementary material provides more details and results that are not included in the main paper due to space limitations. The contents are organized follows:
Section A provides more analysis of aliasing degradation.
Section B compares the aliasing score with confidence-based hard pixel type identification.
Section C discusses the equivalent sampling rate when the kernel size and stride differ in height and width dimensions.
Section provides more implementation details of training.
Section includes an additional ablation study for the Frequency Mixing (Freq.ix) mod- ule.
Section F provides visualization of the frequency response of existing blur filters and demonstrates the advantage of the proposed de-aliasing filter.
Section G provides additional experimental results combined with segmentation boundary refinement methods.
Section H provides an analysis of the orthogonality of downsampling filters.
Section provides a detailed visual analysis, illustrating how aliasing degrades features and leads to three types of errors, and how DAF and FreqMix effectively address aliasing degradation.
Section K analyzes the feature map in the frequency domain.
Section L discusses the aliasing for the transformer-based architecture.
A ALIASING DEGRADATION
We have chosen ResNet (He et al., 2016), Swin Transformer (Liu et al., 2021b), and ConvNeXt (Liu al., 2022) to perform quantitative analysis of the correlation between aliasing scores and errors. Our analysis has been concentrated on the results obtained at object boundaries, where the majority of challenging pixels are found, as mentioned in (Li al., 2017: Gu et al., 2020). ResNet well-established and widely used backbone, whereas Swin Transformer and ConvNeXt represent the latest mansformer-based and CNN- -ased backbones. respectively. Despite the differences in their model structures, our findings, illustrated in Figure 8, reveal a consistent pattern: boundary pixels with higher aliasing scores tend to demonstrate higher cross-entropy errors, indicating that they are more prone to misclassification. These results highlight the pervasive issue of aliasing- induced degradation within modern deep neural networks. This observation underscores the need for comprehensive solutions to mitigate and address this problem, especially in the context of computer vision and other applications where accurate pixel-level information is crucial.
B COMPARING WITH THE PREDICTION CONFIDENCE METHOD
Previous research efforts (Li et al., 2017; Gu et al., 2020) have identified hard pixels based on pre- diction confidence. Here, we compare the prediction confidence method with the aliasing-based method for hard pixel identification. As illustrated in Figure 9, aliasing scores exhibit more con- sistent trend than confidence- based results, especially when considering the effectiveness of distin- guishing three types of errors: displacement errors, false responses, and merging mistakes. This finding underscores the potential of aliasing for categorizing errors effectively.
Notably, these errors display distinct characteristics when analyzed in the context of aliasing, and their importance varies across different scenarios. For instance, displacement emrors tend to cluster in regions with high aliasing scores. In critical applications such robotic surgery radiation therapy, even a slight displacement of just two or three pixels from vital organs like the brainstem the main artery mistakes, which carry greater significance in autonomous driving scenarios, are more commonly found in areas with relatively low aliasing scores.
result in catastrophic consequences. Conversely, false responses and merging
15

---

Published as a conference paper at ICLR 2024
1.4
044
0.2
0.0
(a) ResNet-50
(b) Swin-T
(c) ConyNeXt-T
Figure 8: Illustration depicting the relationship between aliasing scores and hard pixels at boundaries. These
results demonstrate the degradation caused by aliasing affecting three different widely-used models.
Mean error Faise respense
Mean crror of Falso resporse
Mean ertor Marging mistake
Mean ertor of Merging mistake
Displacement
2.00
Entrpy
1.00
1.00
0.50
0.00
0.00
0.20
0.40
1.00
0.25
0.20
0.35
Allasing
Figure 9; Left: Statistically correlated of cross-entropy error for three types of hard pixels with respect prediction confidence (I.i al., 2017). Right: Distribution of the three types of hard pixels.
This analysis not only provides insights into distinguishing between the three error types but also offers valuable guidance for designing error-correcting methods tailored to specific scenarios.
C EQUIVALENT SAMPLING RATE
In the main paper, we provide the calculation of the equivalent sampling rate when the downsam- pling kernel and stride are the same in the height and width dimensions, which the most common situation in existing modern deep neural networks (He et al., 2016; Liu et al., 2021b; 2022; Wang al., 2023). Here, we discuss the equivalent sampling rate when the kernel size and stride in the height and width dimensions are different.
Similarly, we consider both the kernel size and feature size (channel, height, and width), rather than just the downsampling stride. We introduce simple equation for the equivalent sampling rate in the height and width dimensions (ESR ESRW to calculate the actual sampling rate as follows:
Coout
Houl
ESR" H
min(KH down V
X
Cin
Hin
Coout
W
ESR min down>
Cin
Win
where C. H. and W the size of the channel. height, and width. "in" and "out" indicate the input and output features. K H down KV KW present the downsampling kernel size in height and width
dimensions. W foo wea
equal to downsampling stride in height and width dimension, which
16

---

Published as a conference paper at ICLR 2024
Table 8: Ablation study for Frequency Mixing (FreqMix) module.
Method
m
BloUT
BAccf
FEm
MEM,
DEm,
#FLOPs
PPaarms
FregMix w/ decomposition
79.6
58.6
74.7
23.5
52.9
26.3
423.69G
39.64M
FreqMix w/o decomposition
79.7
58.8
74.9
24.0
52.8
26.1
298.67G
31.54M
aligns with (Grabinski et al., 2022). min(K (
down: V Cmn
min( (KW
C C
indicates the influence
C
down'
of the downsampling kernel size Kdown and channel expansion. Notice that we make the common assumption that the impact of channel expansion works for both height and width dimensions, using the square root to calculate the impact for both dimensions.
D MORE IMPLEMENT DETAILS
For the Cityscapes dataset, we employ crop size of 768x 768, a batch size of 8, and a total of 80K iterations. For the PASCAL VOC dataset, we utilize a crop size of 512x512, batch size of 16, and a total of 40K iterations. On the ADE20K dataset, we adopt a crop size of 512x512, a batch size of 16, and total of 80K iterations. We also set the channel of the feature pyramid to 128. We employ stochastic gradient descent (SGD) with momentum of 0.9 and weight decay of 5e-4. The initial learning rate is set at 0.01. During training, we adjust the learning rate using the common 'poly' learning rate policy, which reduces the initial learning rate by multiplying (1 standard data augmentation techniques, including random horizontal flipping and random resizing within the range of 0.5 to 2.
max mier/ )0.9. We apply
For PointRend (Kirillov et al., 2020) and Mask2Former (Cheng et al., 2022) on the LIS dataset, we adopt the same training settings as described in the paper (Chen et al., 2023). We use random flip as data augmentation and train with a batch size of 8, a learning rate of le-2 for 12 epochs, with a learning rate dropping by 10x 8 and 11 epochs, respectively. To make the model quickly adapt to low-light settings, we use COCO pre-trained model as initialization following (Chen et al., 2023).
E ABLATION STUDY
In this section, we present an additional ablation study focused on the Frequency Mixing (FregMix) module. In the main paper, we deseribe our approach predicting weighting values. Specifically, we decompose the prediction of three-dimensional frequency weighting values At, A CR into channel-wise components AJ. AGE # and spatial-wise components At. A #x Where At represents the weighting values for frequencies below the Nyquist frequency, and At represents those above Nyquist frequency. As shown in Table 8, the prediction of three-dimensional frequency weighting values can be computationally intensive for prediction. It directly introduces an additional 7.2 million parameters and requires an extra 125.1 GFLOPs of computational cost. Interestingly, despite this increase in complexity, the overall results are largely similar to the decom- position solution. This highlights the efficiency of our proposed method.
F VISUALIZATION OF VARIOUS BLUU FILTERS
For better comparison, we also illustrate the visualization of the frequency response of existing blur filters, including Blur (Zhang, 2019), AdaBlur (Zou et al., 2020), FLC (Grabinski et al., 2022), and our proposed methods Figure 10. The top row shows the frequency response in two dimensions. We shift the low frequency to the center, and the four corners indicate high frequency, with brighter areas representing higher response. The bottom row displays the frequency response in one dimen- sion, where the left side represents lower frequency, and the right side represents higher frequency. The red line indicates the Nyquist frequency. We observe that Blur (Zhang, 2019) and AdaBlur (Zou et al., 2020) cannot entirely eliminate frequencies higher than the Nyquist frequency. Conversely, due to an underestimation of the Nyquist frequency, FLC (Grabinski et al., 2022) excessively re- moves frequencies below the Nyquist frequency, resulting in information loss. In contrast, proposed de-aliasing filter effectively and precisely removes the frequency power above the Nyquist frequency, which explains its effectiveness.
17

---

Published as a conference paper at ICLR 2024
33
2.1
(a) Blur
(b) AdaBlur
FLC
Ours
Visualization of the frequency response of existing blur filters, including Blur (Zhang, 2019), AdaBlur (Zou et al. 2020), FLC (Grabinski et al., 2022), and proposed methods. The top shows the frequency response two dimensions. We shift the low frequency to the center, and the four corners indicate high frequency, with brighter areas representing higher response. The bottom row displays the frequency response in one dimension, where the left side represents lower frequency, and the right side represents higher frequeney. The red line indicates the Nyquist frequency.
Figure 10:
Table 9: Combination with boundary refinement methods the Cityscapes (Cordts et al., 2016) validation set. Results are reported from the original paper (Yuan et al., 2020).
Method
Doplabb'
(Chen
+GUM
(Muzzini, 2018)
(Kriheebihl) 19.7
DDnnel RF
Seggix
+SogFix+Ouus
mloU
19.3
793
80.3
S1.1
Table 10: Orthogonality analysis for downsampling filters in ResNet (He et al., 2016), Swin Trans- former (Liu al., 2021b), ConyNeXt (Liu et al., 2022), HorNet (Rao al., 2022), and DiNAT (Has- sani & Shi, 2022). Their weights are obtained by training on ImageNet. A higher absolute cosine similarity value indicates greater similarity in filter weights, suggesting lower degree of orthogo- nality (0.0 totally orthogonal, 1.0 = identical filters).
Model
ResNet-18
ConvNeXt-T
Swin-T
HorNet-T
DINAT-L
Abs. CosSim.
0.067
0.072
0.06
0.069
0.046
G COMBINATION WITH BOUNDARY REFINEMENT METHODS
In this section, integrate our proposed method with SegFix (Yuan et al., 2020), a previously effective approach for semantic segmentation boundary refinement. SegFix significantly improves segmentation results by refining predictions, particularly at boundaries where most hard pixels cur. Our proposed method operates independently from SegFix, enhancing models by optimizing intermediate features, precisely removing frequencies leading to aliasing (via the de-aliasing filter), and adjusting frequencies using the encoder block (Frequency Mixing Module).
The synergy between these techniques is evident in the results presented in Table 9, where method enhances SegFix by an additional 0.6 mloU. This improvement highlights the effectiveness of our approach in addressing complex segmentation challenges.
H DOWNSAMPLING FILTERS ORTHOGONALITY ANALYSIS
The introduced calculation of the equivalent sampling rate is based on the assumption that down- sampling filters are orthogonal. In this section, we quantitatively analyze the orthogonality degree of downsampling filters in widely used models, including ResNet (He al., 2016), Swin Trans- former (Liu et al., 2021b), ConvNeXt (Liu et al., 2022), HorNet (Rao et al., 2022), and DiNAT (Has- sani & Shi, 2022). Their weights obtained by training ImageNet.
As shown in Table 10 and Figure 11, we use the absolute cosine similarity as the quantitative mea- surement of the orthogonality degree. A higher absolute cosine similarity value indicates greater
18

---

Published as a conference paper at ICLR 2024
ResNet
ConvNeXt
x
t
Swin
HorNet
15
DINAT
14
Orthogonality degree analysis filters for downsampling in ResNet (He et al., 2016), Swin Transformer (Liu et al., 2021b), ComnNeeteitep2222oonnnen HorNet(Rao et al., 2022), and DiNAT (Hassani & Shi, 2022). We illustrate the absolute cosine similarity matrix, where each element indicates the absolute cosine similarity between different filters. A brighter color indicates higher similarity in filter weights, suggesting lower degree of orthogonality. We observe the matrix showing dark colors, with bright colors only appearing along diagonal (self-to-self), indicating that filters are essentially orthogonal.
Figure 11:
similarity in filter weights, suggesting a lower degree of orthogonality. As depicted Table and Figure 11, the quantitative measurements indicate that the downsampling filters are predominantly orthogonal (with an average absolute cosine similarity ranging from 0.046 to 0.072), thereby sup- porting the introduced equivalent sampling rate in Section C. The proposed equivalent sampling rate designed as heuristic for selecting the cutoff frequency, and we think that exploring how to finely adjust the equivalent sampling rate based the orthogonality of the filter is very interesting and important problem that is worth further investigation.
VISUALIZED ANALYSIS FOR DAF AND FREQMIX
To investigate how the proposed DAF and FreqMix address aliasing degradation and enhance deep neural networks, we visualize the deep features in the model and randomly select some examples in Figures 12 and 13. Furthermore, in Figures 14, visualize three types of errors and demonstrate how the proposed DAF and FreqMix alleviate these errors: 1) false responses, 2) merging mistakes, and 3) displacements.
19

---

Published as a conference paper at ICLR 2024
Image Ground truth
(b) Original
(c) 2x downsampled
(d) 2x downsampled (c) 4x downsampled
( 4x downsampled
Feature
feature with a box. Without DAF in (c) and (e), the downsampling of features exhibits severe "jagged" phenomenon (Zou et al., 2020: Qian al.. 2021), resulting in the degraded representation of object boundaries. The response of some objects is faded lost in the 4x scale in (c). By directly removing the high frequency leads to aliasing degradation, the proposed DAF can largely relieve the "jagged" phenomenon (Zou et al., 2020; Qian 2021), making the boundaries more clear the (d) and (f). Furthermore, DAF largely preserves the object responses, as shown in (I), compared to (e).
Figure 12: Visualization for We mark some high aliasing score areas
Heatmap for low frequeeyy
(b) Heatmap for high-fieeuency
Feature map wia F'regMix
(d) Feature map w/ FreqMix
Feature frequency
(f) Feature frequency wi FregMix
wia FrreMix
Figure 13: Visualization for FreqMix. (a) and (b), the heatmap shows brighter color object boundaries and a darker color for object centers and backgrounds, especially for high gigh-frequency components, where brighter color indicates a high value. Thus, FreqMix not only reduces the overall high high-frequency content responsible for aliasing degradation in (f) but also preserves the high frequency of object boundaries. This preservation is crucial for making the boundaries clear in (d), ensuring accurate segmentation, and lowering the occurrence of three types errors.
20

---

Published as a conference paper at ICLR 2024
(9)
(c)
(8)
(c)
(k)
w/o DAF feature
w/ DAF feature
w/o DAF prediction
w/ DAF prediction
Ground truth
wio FreqMix feature
FreqMix w/o FreqMix
FreqMix
Ground truth
f'eature
prediction
prediction
Visualization of how the proposed DAF and FreqMix address aliasing degradation and im- prove the feature and final segmentation. We randomly select image patches with high aliasing score from Cityscapes (Cordts et al., 2016) dataset validation set. Zoom for better view.
Figure 14:
I.1 VISUALIZATION FOR DE-ALIASING FILTER (DAF)
As depicted in Figures 12(c) and (e), we indicate some areas with a high aliasing score in the fea- ture with the red box, they exhibit a severe "jagged" phenomenon (Zou et al., 2020; Qian et al., 2021), leading to the degraded representation of object boundaries. Moreover, in comparison with the original feature
Figure 12(b), the response of some objects is lost in the 4x downsampling in Figures 12(e). It is noteworthy that widely used state-of-the-art models, such as Swin Trans- former (Liu et al., 2021b) and ConvNeXt (Liiu al., 2022), adopt a total downsampling stride of 32x, potentially leading to even more severe loss of response.
By directly removing the high frequency leads to aliasing degradation, the proposed DAF can largely relieve the "jagged" phenomenon (Zou et al., 2020: Qian et al., 2021), making the boundaries more clear the Figures 12(d) and (f). Furthermore, DAF largely preserves the object responses, as shown in Figure 12((), compared to Figure 12(e), resulting in more accurate segmentation prediction and a lower occurrence of the three types of errors shown in the left column Figure 14.
1.2 VISUALIZATION FOR FREQUENCY MIXING MODULE (FREQMIX)
FreqMix improves the model by decomposing features into low frequency and high frequency using a Nyquist frequency threshold and dynamically selecting them in a spatial-variant manner. We vi- sualize the heatmap for selecting low frequency and high frequency Figures 13(() and (b), where a brighter color indicates a high value. The heatmap shows a brighter color for object boundaries and a darker color for object centers and backgrounds, especially for high-frequency components. Thus, FreqMix not only reduces the overall figh-freyuency content (see Figure 13(e) and (f), re- sponsible for aliasing degradation, but also preserves the high frequency of object boundaries. This preservation is crucial for making the boundaries clear (see Figure 13(c) and (d)), ensuring accu- rate segmentation and a lower occurrence of three types of errors. Further visualization in the right column Figure 14 verifies that Freq reduces the occurrence of the three types of emros..
21

---

Published as a conference paper at ICLR 2024
w/o DAF
w/ DAF
w/o FreqMix
w/ FreqMix
Visualization of the averaged frequency distribution of features. The center indicates low fre- quency, and the comners indicate high frequency. brighter color indicates more corresponding frequency components. The DAF directly removes the frequency above the Nyquist frequency, while FreqMix suppresses the high frequency.
Figure 15:
10a
FreqMix
Frequency amplitude
A Frequency amplitude
10a
w/ Freq
Nyquist frequency
10
10
10a
10-3
w/o DAF w/ DAF
10a
10
Nyquist frequency
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Low frequency -
High frequency
Low frequency
- High frequency
DAF
(b) PreqMix
Figure 16:
Visualization of the frequency distribution of features.
I.3 VISUALIZATION OF THREE TYPES OF ERRORS
As illustrated in Figure 14, randomly selected image patches with a high aliasing score from the Cityscapes (Cordts et al., 2016) dataset validation set.
We observed that aliasing leads "jagged" phenomenon (Zou al., 2020; Qian al., 2021), disrupting object shapes and boundaries, resulting in false responses (Figures 14(b) and (I)) and displacement errors (Figures 14(c), (d), and (e)). DAF and FreqMix relieve this phenomenon and improve the feature representation thus resulting in lower false responses and displacement errors.
When high-frequency information is aliased, it transforms into false low-frequency information. For example, when two objects are close to each other, their high-frequent boundaries can be aliased lower frequency during downsampling, causing the two objects appear connected and their boundaries to be merged. This leads to merging errors (Figures 14(a), (g), and (j)). DAF/FreqMMx solve this by removing/suppressing these high frequencies during downsampling/encoder block, leading to lower merging errors.
Moreover, high-freyuency components in the object center or background can result in false re- sponses (Figures 14(i) and (k)). FreqMix addresses this issue by suppressing the high frequency in the object center or background while preserving the high frequency the boundaries.
J FEATURE FREQUENCY ANALYSIS
We present feature frequency analysis in Figures 15 and 16. The DAF directly eliminates fre- quencies above the Nyquist frequency, while the FreqMix suppresses high-frequency components, alleviating aliasing degradation.
22

---

Published as a conference paper at ICLR 2024
K FEATURE FREQUENCY ANALYSIS
We present a feature frequency analysis Figures 15 and 16. The DAF directly eliminates fre- quencies above the Nyquist frequency, while the FreqMix suppresses high-frequency components, alleviating aliasing degradation.
L DISCUSSION ABOUT ALIASING IN A TRANSFORMER-BASED ARCHITECTURE
As for recent transformer-based architectures, aliasing remains concern. Taking the renowned Vision Transformer (ViT) as an example, ViT (Dosovitskiy et al., 2020) tokenizes images by splitting them into non-ovenn overlapping patches, which are then fed into transformer blocks. The tokenization and self-attention operations performed on these discontinuous patch embeddings can be viewed as downsampling operations, introducing a potential side effect aliasing. essential to note that this downsampling operation is virtually unavoidable due to the spatial redundancy nature of the image (He et al., 2022) and huge computational costs without downsampling (increasing by 256x without downsampling in ViT).
Several existing studies have acknowledged this concern. A straightforward solution to alleviate aliasing is to increase the sampling rate. Similar trends are observed in vision transformers, where the use of overlapped tokens (Yuan et al., 2021) and smaller patch sizes (Caron et al., 2021) con- tributes to improved performance. However, escalating sampling rates incur quadratic computational costs. Consequently, hypothesize that integrating appropriate anti-aliasing filters into the 'attend- ing' process could offer viable solution. In fact, existing work has empirically explored blending anti-aliasing filters into the vision transformer, reporting observed improvements (Qian et al., 2021).
In conclusion, aliasing persists potential concern in transformer-based architectures, and prior studies have endeavored to address it empirically by enhancing the sampling rate or integrating anti- aliasing filters into the attention mechanism. Our work represents step forward in quantitatively assessing and addressing aliasing in contemporary models, supported by theoretical foundations. This issue still presents ample opportunities for further investigation and exploration.
23


Reviews:
[{"review_id": "Hifo7NSJtL", "reviewer": "Reviewer_qrZx", "summary": "The paper proposes two novel de-aliasing filter (DAF) and frequency mixing (FreqMix) modules to alleviate aliasing degradation by accurately removing or adjusting frequencies higher than the Nyquist frequency. The paper observes three different wrong segmentation types potentially caused by aliasing (a) False response, (b) Merging mistake (c) Displacement. In addition, the paper designed a simple de-aliasing filter to precisely remove aliasing as measured by their aliasing score. Additionally, we propose a novel frequency-mixing module to dynamically select and utilize both low and high-frequency information. The work has really comprehensive ablation studies, and the induction logic is comprehensive. They have proved their proposed method surpasses strong baselines Mask2Former, PointRend.", "questions": "It would be really appreciated if the authors shared their thoughts on whether aliasing is still a problem if the backbone downsampling is fully adaptive in a transformer-based architecture.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The experiment of the paper is very comprehensive and solid. The authors provide analysis of (1) the relationship between blur kernel size with Boundary, Three type errors, and Aliasing score.  (2) Noise level in effect to aliasing. (3) their cut-off frequency in relationship with accuracy and aliasing. (4) In comparison with other anti-aliasing modules. (5) Show effectiveness with model scaling up. (6) In Comparison with SoTA segmentation model.\n\n2. The paper introduces the concept of equivalent sampling rate for the Nyquist frequency calculation and proposes an aliasing score for quantitative measurement of aliasing levels.", "weaknesses": "Actually, I think from a research perspective, I believe the paper is valid and sound so I recommend accepting this paper. However, from a higher level of view, many traditional problems including anti-aliasing have been eroded by the current trend of Large Models. The effectiveness of more training data or advances in pre-trained model weights will shrink the marginal gain of those methods. Especially for the transformer-based architecture, the downsample operation is no longer max-pooling even adaptive-pooling. this potentially alleviates the aliasing problem itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TC5pfRQVWO", "reviewer": "Reviewer_LE8t", "summary": "The work proposes a new anti-aliasing scheme by redefining the cutoff frequency. This is done by considering the expansion in both the channel dimension and the spatial dimension. The proposed aliasing score showed a strong correlation with various segmentation errors. Additionally, the authors introduced a new anti-aliasing and spectral filter that enhances the segmentation performance.", "questions": "1. I need more clarification about the robustness of ESR (equivalent sampling rate). Clearly, the filters are orthogonal for the shown example (Fig 3). So there is no loss of information.\nBut what if the filters are not orthogonal to each other? Assume a worst-case scenario where all filters are equal. In that case, the proposed ESR will give us a wrong sampling rate. So, is the proposed ESR intended to replace the regular “sampling rate,” or is it intended as a heuristic to select cutoff frequency? If it is the second, then the paper should clarify it.\n2. Section 3.2, “Metrics for three errors.” — I think the middle one should be the definition of MErr.\n3. “aligning with the observations in Figure 1 that false responses and merging mistakes predominantly exist in areas with relatively low aliasing scores.” — could you clarify how we are associating aliasing scores with a region of an image? \n4. “However, for deeper features at the second and third stages, the aliasing ratio decreases.\nThis decrease does not signify a reduction in aliasing; rather, it occurs because the earlier high level of aliasing results in severe degradation. Consequently, the features become ‘hard pixels,’ meaning that the following stage struggles to extract useful information and loses response to objects.” — How can we confirm that it is caused by the side effect of aliasing? Applying significant noise can generate an out-of-distribution (OOD) sample of the internal layers, which can also lead to poor feature extraction.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. Investigates the critical yet under-explored question regarding the effect of aliasing in computer vision models.\n2. The proposed fixes enhance performance.\n3. The paper is well-written and clearly explained.", "weaknesses": "1. The proposed module, FreqMix, requires additional forward and inverse Fourier transforms at each layer. Compared to other methods, this increase in time complexity (inference and training) needs to be discussed.\n2. Experiments are limited. It is necessary to conduct evaluations on other benchmark datasets, such as MS COCO and Pascal VOC.\n3. Improvements are marginal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VXAaDGF4hp", "reviewer": "Reviewer_uz5r", "summary": "In this paper, the authors introduce a challenging yet critical topic for semantic segmentation, i.e., pixel-wise aliasing. They categorize the hard pixel error into three types: false response, merging mistakes and displacements. Creatively, the de-aliasing filter and frequency mixing modules are introduced to alleviate the aliasing degradation. Experiments demonstrate that these findings can consistent improve the semantic segmentation performance.", "questions": "#1. The authors must provide more visualizations on the three types of errors and the corresponding feature maps in frequence domain. \n#2. As to the DAF and FreqMix, how the filtered feature maps can help find the location of error map should be invastigated. \nOverall, the topic for finding the aliasing is interesting, the reviewer encourage the author further explore this missing parts in the semantic segmentation.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "#1 The topic is attractive where the unexplored aliasing phenomenon for semantic segmentation are carefully investigated. This funding can motivate the researcher in the related community. \n#2 The authors utilize the DAF and FreqMix to remove the aliasing in Fourier domain and balance high-frequency components in the encoder block. Converting the features to frequency domain by DFT is straightforward, but finding the aliased false prediction for semantic segmentation is interesting.", "weaknesses": "#1 Lacking visualization for the three type of errors in the experiment. Though the authors provide the results on ADE20K, it does not clearly present the visualization to the correction of three type of errors for semantic segmentation. In addition, how the DAF and FreqMix help locate these errors are not visualized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)