Title: Soft Superpixel Neighborhood Attention
Venue: NeurIPS 2024
Accepted: yes
Keywords: attention module

superpixels

image denoising

deep learning
Abstract: Images contain objects with deformable boundaries, such as the contours of a human face, yet attention operators act on square windows. This mixes features from perceptually unrelated regions, which can degrade the quality of a denoiser. One can exclude pixels using an estimate of perceptual groupings, such as superpixels, but the naive use of superpixels can be theoretically and empirically worse than standard attention. Using superpixel probabilities rather than superpixel assignments, this paper proposes soft superpixel neighborhood attention (SNA), which interpolates between the existing neighborhood attention and the naive superpixel neighborhood attention. This paper presents theoretical results showing SNA is the optimal denoiser under a latent superpixel model. SNA outperforms alternative local attention modules on image denoising, and we compare the superpixels learned from denoising with those learned with supervision.

Soft Superpixel Neighborhood Attention

Abstract

Images contain objects with deformable boundaries, such as the contours of a human face, yet attention operators act on square windows. This mixes features from perceptually unrelated regions, which can degrade the quality of a denoiser. One can exclude pixels using an estimate of perceptual groupings, such as superpixels, but the naive use of superpixels can be theoretically and empirically worse than standard attention. Using superpixel probabilities rather than superpixel assignments, this paper proposes soft superpixel neighborhood attention (SNA) which interpolates between the existing neighborhood attention and the naive superpixel neighborhood attention. This paper presents theoretical results showing SNA is the optimal denoiser under a latent superpixel model. SNA outperforms alternative local attention modules on image denoising, and we compare the superpixels learned from denoising with those learned with superpixel supervision. 1

1 Introduction

The attention mechanism is attributed to meaningful benchmark improvements in deep neural networks [1]. While the initial attention mechanism acted globally, recent methods operate on local neighborhoods, which requires less computation and reduces the chance of learning spurious correlations [2]. Neighborhood attention (NA) transforms each pixel using its surrounding square neighborhood of pixels. In reality, this square neighborhood is merely an implementation convenience. Natural images contain objects with non-rigid boundaries, such as the contours of a human face or the outlines of text. To account for these deformations, this paper proposes a new local attention module that re-weights the attention map according to low-level perceptual groups, or 'superpixels' [3], named soft superpixel neighborhood attention (SNA).

Theory from vision science posits that humans view scenes according to perceptual groups (e.g. Gestalt principles) [4]. Superpixels are one such formulation of perceptual groupings, and they create a deformable, boundary-preserving segmentation of an image [3]. Since deep learning models can learn spurious correlations, operations that utilize perceptual groups should principally help a network learn desirable correlations. Despite the deformable neighborhood shapes, applying attention with a superpixel estimate is straightforward. However, this naive approach is theoretically worse than NA for some superpixel shapes and sensitive to errors in the superpixel estimate. To address these issues, we propose thinking beyond using a single superpixel estimate.

While recent works use the maximum likelihood (ML) superpixel estimate [6, 7], many superpixel formations are equally likely. Figure 1 depicts three samples from a (slightly modified) recent BASS superpixel model [5]. Many of the boundaries seem arbitrary and change among the samples, while some remain fixed. No single superpixel segmentation is the absolute 'best'. This is not a deficiency of the particular method to estimate superpixels, but rather a drawback of using a point estimate.

Originallmage

ThreeSuperpixel Estimates

DifferentSuperpixels

This paper presents a theoretically grounded approach to account for the ambiguity of superpixel assignments within an attention module. We consider this ambiguity in our derivation by re-weighting the empirical data distribution with superpixel probabilities. While standard superpixel models assume a fixed prior probability, this paper supposes each pixel has a separate prior superpixel probability. Under this new model, we find the optimal denoiser is the proposed SNA module.

Contributions. In summary, this paper presents a rigorous way to incorporate superpixels into an attention module. By modeling image pixels with distinct superpixel probabilities, we find the soft superpixel neighborhood attention (SNA) module is the optimal denoiser. We empirically show the SNA module outperforms NA within a single-layer denoising network by 1 -2 dB PSNR. We compare the superpixels learned for denoising with superpixels learned using supervised training [8].

2 Related Works

Non-Local Methods. Classical image restoration methods transform a pixel using its neighborhood based on pairwise similarities [9-11]. Some early deep learning models incorporate non-local modules with deep learning models for image restoration [12-17]. Once attention was introduced to the computer vision community, later methods drew analogies to between attention and non-local methods [18-20].

Attention. While attention modules followed the same form as non-local methods, they were originally presented as a disconnected idea [1, 21]. Attention's primary issue was originally computation, and subsequent research efforts proposed efficient alternatives [22-25, 2]. These alternatives often compute attention across a smaller region, and use a deep network's depth to disperse information globally. Neighborhood attention (NA) is implemented with an efficient, custom CUDA kernel to search a square window and is among the fastest methods [2]. The square window is theoretically more desirable than asymmetrical alternatives, such as SWIN [22].

Superpixels. Superpixels embody the concept of perceptual groupings, which suggests that humans first group smaller objects together and then aggregate this information to understand the whole image [3]. There are many methods to model these superpixels [26, 27]. SLIC is a state-of-the-art, classical superpixel method and is explained in Section 3.1 [26]. Superpixels have been applied extensively outside of deep learning literature [28-32]. Superpixels have not been employed as widely since the advent of deep learning, but recent methods have started incorporating them [6, 7]. A recent method, referred to as the superpixel sampling network (SSN), presents a supervised learning method to sample using segmentation labels [8]. The loss function of SSN is related to the idea of superpixel pooling, which we use to evaluate superpixel probabilities in Section 5.3 [33, 34].

Non-Parametric Density Estimation. The derivation of the optimal denoiser using latent superpixels closely resembles the problem of mixture-based non-parametric density estimation [35]. This paper does not address important statistical issues, such as identifiability.

Image

Image

SuperpixelMeans

(Centroids)

H

Superpixel

H

H/Ssp

Center

Ssp

W/Ssp

W

W

3 Preliminaries

3.1 SLIC Superpixels

In this paper, an image ( x ) has height ( H ), width ( W ), and features ( F ) with shape HW × F . Conceptually, superpixels are evenly spaced across the image according to the stride denoted S sp . The superpixel stride ( S sp) determines the number of superpixels, S = HW / S 2 sp (see Fig 2). SLIC superpixels consist of superpixel means ( ˆ µ with shape S × F ), intra-superpixel precision ( ˆ η = [ ˆ η app ˆ η shape ] with shape S × 2 ), the probability each pixel belongs to each superpixel ( ˆ π with shape HW × S ), and the superpixel assigned to each pixel ( ˆ s with shape HW ). SLIC estimates these quantities using Lloyd's algorithm (similar to k-means) [26]. Following recent works, each pixel is connected to at most nine superpixels [8].

Presently, we describe how SLIC superpixel probabilities are computed. Let the distances between pixels and superpixel means be written as a matrix D of shape HW× S and let D ( i,s ) = ∥ x i -µ s ∥ 2 2 if the two are connected and infinity otherwise. A difference between the indices is written ˜ D ( i,s ) = ∥ [ i x i y ] -[ s x s y ] ∥ 2 2 . Superpixel probabilities are computed as,

where ˆ η i, app , ˆ η i, shape > 0 enforce consistency across appearance ( app ) and shape . The estimated superpixel probabilities are the conditional probability pixel i is assigned to superpixel s , p ( s i = s | x i ) = ˆ π ( i,s ) . A superpixel segmentation ( ˆ s ) is formed by assigning each pixel's cluster label to their most similar superpixel; ˆ s i = arg max s ˆ π ( i,s ) . These quantities are estimates (note the ̂ hats) of latent parameters, which is a modeling assumption discussed in Section 4.2.

3.2 Superpixel Pooling

The attention module proposed in this paper will learn superpixel probabilities, but standard superpixel evaluation only considers a point estimate. To augment the standard superpixel benchmarks, we follow recent works and compare superpixel probabilities with superpixel pooling [8, 5, 34]. A matrix z has size HW× C for C segmentation classes, and superpixel probabilities ( ˆ π ) have size HW× S . Then, the probabilities are re-normalized, ˜ π ( i,s ) = ˆ π ( i,s ) / ∑ HW j =1 ˆ π ( j,s ) . Finally, the superpixel pooling operation is written,

In this paper, the vector z is either an image ( C = F is the number of channels) or a segmentation label. A recent method, named the Superpixel Sampling Network (SSN), proposes learning taskspecific superpixels with this superpixel pooling loss [8]. Their loss function trains a network to estimate superpixel probabilities by comparing the original and pooled vectors. In Section 5.3, we compare the quality of superpixels learned within an SNA-denoising network and those learned using the SSN loss on the BSD500 dataset [36].

Neighborhood

HardSuperpixel

SoftSuperpixel

Attention (NA)

Neighborhood

Neighborhood

Attention (H-SNA)

Attention (SNA)

3.3 Neighborhood Attention

Let a noisy input image be denoted x with shape HW × F . The queries ( q ), keys ( k ), and values ( v ) project the image x from dimension F to D , written as q = xW q , k = xW k , v = xW v . Neighborhood attention (NA) computes the output of the attention operator using pixels within a square neighborhood around pixel i , denoted N ( i ) [2]. A standard neighborhood of size 7 × 7 is among the fastest available attention methods. The attention scale controls the shape of the attention weights, λ at ≥ 0 . The output and attention weights are computed below,

As an additional baseline, this paper augments the standard NA by learning the attention parameter for each pixel. Specifically, each pixel's attention scale is the output of a deep network, λ ( i ) at = g ( i ) ϕ, Deep ( x ) . Further details are described in Section 4.5.

4 Approach

4.1 Superpixel Neighborhood Attention

̸

Hard Superpixel Neighborhood Attention (H-SNA). Perceptual groupings, such as superpixels, segment pixels into a boundary-preserving partition of the image. These boundaries account for sharp changes in pixel intensity, so using these segmentations as masks can remove perceptually unrelated information. Supplemental Section A.2 formalizes this intuition. H-SNA is a naive implementation of this concept. For a fixed query pixel ( i ), a neighboring pixel ( j ) is masked if its estimated superpixel does not match the query's superpixel, ˆ s i = ˆ s j . The function is written as,

Soft Superpixel Neighborhood Attention (SNA). There are two problems with H-SNA. First, H-SNA is unstable when the number of pixels sharing the same superpixel label within the local neighborhood is small. Examples include snake-like tendrils and superpixels that consist of small, disconnected regions (see Section A.2). The second problem with H-SNA is its high sensitivity to improper superpixel estimation. Erroneous ML estimates of superpixels have a dramatic effect on quality. To mitigate both issues, we propose soft superpixel neighborhood attention (SNA), which uses superpixel probabilities instead of superpixel assignments,

π~p(π)

S~ Cat(π(i)

x~ p(x|s)

Step 1

Step 2

Step 3

Importantly, the sum over all superpixels contains only 9 non-zero terms (see Section 3.1). The shape of the superpixel probabilities allows SNA to interpolate between NA and H-SNA. To be specific, the superpixel probabilities may be flat to match NA, ˆ π ( i,s ) ≈ 1 9 , or the superpixel may be sharp to match H-SNA, ˆ π ( i,s ) ≈ 1 [ s = s ∗ ] . This control over the shape of the superpixel probabilities allows SNA to be better than both H-SNA and NA on average.

4.2 The Latent Superpixel Model

Properly incorporating superpixels into attention requires rethinking the relationship between superpixels and images. While superpixels are usually presented with a generative model for images, this model is used only to estimate superpixel assignments. In contrast, this paper uses the data-generating process itself to derive the form of our proposed attention module. Our process is slightly different from standard literature. Ordinarily, all pixels share a single prior superpixel probability. In this paper, the superpixel probabilities are sampled for each image pixel.

For each image pixel, our generative model samples (1) superpixel probabilities π ∼ p ( π ) , (2) superpixel assignments s i ∼ p ( s i | π ( i ) ) , and (3) pixel values x | s ∼ p ( x | s ) . Figure 4 illustrates this proposed latent variable model. Note that SLIC estimated these quantities (recall the hats: ˆ s , ˆ π ), but assumes a different data generating model. While the distribution of pixels given the superpixel assignments is unknown, the new model is useful for theoretical analysis. We use it to derive the form of our proposed attention module (Sec 4.3), and to compare the theoretical error between SNA, NA, and H-SNA (Sec A.2).

4.3 SNA is the Optimal Denoiser under the Latent Superpixel Model

The optimal denoiser is the function that minimizes the mean-squared error between the denoised pixel and the original pixel value. Following similar derivations [37], minimizing this expectation with respect to an unknown denoiser ( D ) is possible because of our chosen model for the unknown data density. Say we have a sample, x , s , π ∼ p ( x , s , π ) . Then we approximate p ( x i | s i ) ,

since p ( s i ) = 1 9 , and p ( m ) = 1 M . By re-weighting with superpixel probabilities, rather than superpixel assignments, our estimate does not depend on ambiguous superpixel labels. The expected loss of a denoiser is computed over the factorized joint density of the Gaussian-corrupted pixels, noisefree pixels, superpixel assignments, and superpixel probabilities: p ( ˜ x i | x i )ˆ p ( x i | s i ) p ( s i | π ( i ) ) p ( π ) .

Claim 1 The optimal denoiser to the following optimization problem is soft superpixel neighborhood attention (SNA) when the qkv-transforms are identity, the attention scale is fixed to λ at = 1 2 σ 2 , and the samples used to estimate the data density come from a neighborhood surrounding pixel i ,

9Φ,SLIC

gΦ,Deep

Each Superpixel

Probability of

Each Superpixel

Probability of

H

/app

A

H

UNetSoftmax

UNet

ReLU

shape

A

SLIC

H

H

Iterations

H

W

W

M

SLIC

Parameters

W

x5

W

See Supplemental Section A.1 for the proof. In practice, the error of a denoiser depends on the sampled data, and in this paper, the data is limited to a neighborhood surrounding the query point. Section A.2 analyzes this limited neighborhood.

4.4 Normalization for Restricted Connectivity

SLIC [26] and SNN [8] connect each pixel to (at most) nine superpixels. Therefore, some pixels within a neighborhood may only connect to a subset of superpixels that connect to the query index. This sharp cut-off in connectivity creates artifacts within the attention map since the number of non-zero superpixel probabilities for some neighbors is fewer than nine. These artifacts are due to our choice of superpixel algorithms [26, 8] rather than a limitation of our attention module. To correct these artifacts for the attention map associated with pixel i , we re-normalize the adjacent superpixel probabilities to one; π ( j,s ) → π ( j,s ) / ∑ s ′ ∈C ( i ) π ( j,s ′ ) , where C ( i ) is the set of superpixels connected to pixel index i . Figure 5 illustrates the attention maps with and without the normalization step.

4.5 Estimating Superpixel Probabilities

SNA re-weights the attention map using superpixel probabilities, which are estimated from noisy images, π ≈ ˆ π = g · ( x ) . This paper forgoes considering theoretical concerns (such as identifiability) and uses ad-hoc estimates of these probabilities with two deep networks, depicted in Figure 6.

The first method estimates the marginal probabilities directly from a deep network, ˆ π = g ϕ, Deep ( x ) . The second method estimates the superpixel parameters controlling the balance between appearance and shape with a deep network, which is then fed into differentiable SLIC iterations [8] (Section 3.1): ˆ π = g ϕ, SLIC ( x , ˆ η ) where [ ˆ η app ˆ η shape ] = ˜ g ϕ, SLIC ( x ) . This model is abbreviated as ˆ π = g ϕ, SLIC ( x ) . The number of SLIC iterations is fixed to 5 , following a recent paper [7]. These networks are two-layer UNets [38].

4.6 Learnable Attention Scale

Using a network similar to the ones used to estimate superpixel probabilities, all attention modules are augmented to serve as another baseline. The augmentation replaces the fixed attention scale with learnable attention scales for each pixel, λ at = g ϕ, Attn ( x ) . The network architecture is identical to g ϕ, Deep except for the final layer, which outputs one channel rather than nine. This baseline empirically demonstrates the theoretical argument from Supplemental Section A.3; even when modulating the attention scale, NA cannot robustly reject dissimilar pixels.

S

yj=

8=1

5 Experiments

This section demonstrates the impressive benefit of superpixel neighborhood attention compared to standard neighborhood attention. To verify whether the improvement is due to the proposed method, we compare several variations of both attention methods. Section 5.2 compares different attention modules within a simple network architecture on Gaussian denoising, which empirically verifies the theoretical findings in Sections 4.3 and A.2. Section 5.3 compares the superpixel probabilities learned from the denoising loss function with superpixels learned through supervised training. Supplemental Section B.1 includes ablation experiments.

5.1 Experimental Setup

Our experiments use a simple denoising network with larger auxiliary networks to highlight the role of the learned superpixel probabilities and attention scale parameters. The denoising network contains only projection layers and a single attention module. To keep the extracted features simple, the only interaction between neighboring pixels is through the attention layer. The project matrices project the input RGB image from three to six dimensions dimension. The full network is written,

The primary network parameters are denoted θ = [ W 0 , W 1 ] , and g ϕ ( · ) is the auxiliary deep network from Section 4.5. The primary network consists of only 200 parameters ( θ ), and the auxiliary networks contain about 4 . 4 k or 8 . 8 k parameters ( ϕ ). We train each network for 800 epochs using a batch size of 2 on the BSD500 dataset [36] using a learning rate of 2 · 10 -4 with a decay factor of 1 / 2 at epochs 300 and 600. The network is optimized with Adam [39]. The code is implemented in Python using Pytorch, Numpy, Pandas, and CUDA and run using two NVIDIA Titan RTX GPUs and one RTX 3090 Ti GPU [40-43]. Testing datasets are Set5 [44], BSD100 [36], Urban100 [45], and Manga109 [46].

5.2 Gaussian Denoising

This subsection empirically verifies the utility of SNA on Gaussian denoising. The Charbonnier loss is used for denoiser training, L Deno = √ ∥ ˆ y Deno -y ∥ 2 + ε 2 ( ε = 10 -3 ) [47]. By default, the window size is 15 × 15 , the fixed SLIC precision is ten, and the fixed attention scale is one.

Table 1 presents the quantitative results. The SNA module achieves far better denoising quality than NA, even when NA can learn its attention scale. As an extreme example, the PSNR at σ = 30 of the left-most SNA column is about 2.8 dB better than standard NA (2nd NA column). If comparison is restricted to models with the same number of parameters, the PSNR at σ = 30 of the the fourth SNA column is about 1.8 dB better than the augmented NA (1st NA column). For a fixed budget of network parameters, SNA yields a higher quality denoiser than NA.

Figure 7 shows SNA is qualitatively superior to NA. The first row shows that SNA produces less grainy images than NA. The bottom two rows show that NA mixes perceptually unrelated information. In the second row, NA outputs a red-orange semi-transparent mist surrounding the orange-red rock. In the third row of the NA results, the white zebra stripes are shaded darker than the clean image when the attention scale is learned, and contain white speckles if fixed. See Section B.2 for more examples.

Learning the attention scale improves standard NA and SNA+ g ϕ,Deep at higher noise intensities. However, it degrades model quality when superpixels are estimated with SLIC iterations (SNA+ g ϕ,SLIC ). Perhaps this quality drop is simply due to network architecture and/or the flow of information. Since the network that learns the attention parameter is only given a noisy image, the simple model struggles to coordinate with the denoising network, which uses multiple SLIC iterations. SLIC iterations explicitly compute pairwise distances, which cannot be learned with the shallow model.

A limitation of the proposed SNA module is the additional computation compared to NA. SNA amounts to re-weighting the attention map produced by a neighborhood search, so SNA is necessarily more compute-intensive than NA. The bottom four rows of Table 1 show SNA is 15 - 22 times slower than NA. While this increase is significant, we believe the numbers reported in this paper are overly pessimistic. Our implementation of SNA has not been optimized, and future development can dramatically reduce the wall-clock time. One way to reduce wall-clock time is to read more efficiently

SNA

NA

Noisy

Clean

Learn 入at

Fix 入at

Learn 入at

CastCasiCastCasicast

22.09

27.19

28.15

25.36

24.81

22.07

27.17

27.29

24.61

25.75

22.63

27.38

28.25

21.89

25.64

from global CUDA memory [43]. Anecdotally, the naive implementations of NA are 500% to 1500% slower than their recently optimized alternatives.

5.3 Inspecting the Learned Superpixel Probabilities

This subsection investigates how the superpixels learned from denoising compare with superpixels learned from explicit supervised superpixel training [8]. We observe a collaborative relationship between superpixel probabilities for denoising and superpixel pooling. We observe an adversarial relationship between the superpixels for denoising and boundary adherence. Generally, learning superpixels that improve superpixel benchmarks decreases their utility for denoising.

To compare the different training regimes for learning superpixel probabilities, an SNA network is fully trained with a combination of two loss functions. One loss is the denoising loss ( L Deno). The second loss is the SSN loss function, which computes the difference between a target vector and its superpixel-pooled transformation (see Section 3.2 for super-pixel pooling) [8]. Specifically, L SSN = L (ˆ z sp-pool , z ) where L is the cross-entropy loss for a segmentation label and the meansquared-error for image pixels. The superpixel stride for all methods is fixed to S sp = 14 . Each network estimates superpixels using SLIC iterations, and the attention scale is fixed to one (if applicable). Evaluation is computed on the test set of BSD500 [36].

The Achievable Segmentation Accuracy (ASA) and Boundary Recall (BR) scores evaluate the quality of the maximum likelihood superpixel assignment. The ASA score measures the upper bound on the achievable accuracy of any segmentation method, and the BR measures the boundary alignment between the superpixels and a segmentation. The superpixels are processed with a connected components algorithm before computing ASA and BR scores. To qualitatively compare the learned superpixel probabilities, we follow recent literature and compare the superpixel pooled images [5]. Images are compared against their superpixel pooled images using PSNR and SSIM.

Table 2 evaluates the denoising quality, Table 3 quantitatively evaluates the superpixel quality, and Figure 8 qualitatively evaluates the superpixel pooling quality. We observe a harmony between the denoising loss and pixel pooling loss terms. Combining the two terms yields slightly improved denoising, as reported in the first two rows of Table 2. The first, third, and fourth columns of Table 3 show the ML superpixel quality is similar, but the boundary recall is slightly worse when combining the two. The superpixel pooling quality increases by 2.33 dB PSNR when combined compared to only denoising. The second and last columns of Figure 8 qualitatively compare the pooled images.

We observe a disharmony between the superpixel probabilities useful for denoising and those trained on segmentation labels. Table 2 reports a 2.6 dB PSNR drop in denoising quality . The first two rows of Table 3 report training with segmentation labels yields the best ML superpixel estimates, matching related works [8]. However, the superpixel pooling quality is poor. This suggests boundary detection is distinct from denoising and pixel pooling.

SSNLabel

Segmentation

ImagePixels

LoSS

LDeno

LsSN

LsSN

LsSN

LsSN + LDeno

10

0

0

10

10

199893

98893

28.85

24.05

31.63

30.63

30.64

29.00

24.79

31.99

31.09

30.90

TrainedforImage

5.4 Comparing Computational Complexity

Table 4 compares the FLOPs and memory consumption of SNA and NA from Equations 3 and 5. Let the neighborhood window size be written |N ( i ) | = K 2 . In summary, the FLOPs estimate for SNA and NAis O ([2 F +10] ·HW K 2 ) and O ([2 F +1] ·HW K 2 ) , respectively. The peak memory consumption for SNA and NA is O ( HW F +2 HW K 2 ) and O ( NF + HW K 2 ) , respectively. Importantly, SNA's additional complexity does not scale with the number of features (F). Since the number of features is a significant factor in the computational cost of an attention operator, it is sensible to believe SNA can be used efficiently within large-scale deep neural networks. To be concrete, if the number of features is 128, then there is less than a 4% percent increase in FLOPs from NA to SNA.

6 Discussion

This paper presents the soft superpixel neighborhood attention (SNA) module as an alternative to neighborhood attention (NA). SNA accounts for deformable boundaries of objects. The key modeling assumption is that superpixel probabilities vary per pixel, and we find SNA is the optimal denoiser under this model. For a fixed budget of network parameters, SNA achieves a significantly better denoising quality than NA within a single-layer neural network. While SNA does require more computation than NA, SNA's computation does not scale with features which suggests SNA can be used efficiently within large-scale deep neural networks.

7 Acknowledgments

This work is supported, in part, by the National Science Foundation under the awards 2030570, 2134209, and 2133032. The authors thank Drs. Vinayak Rao and David Inouye for their valuable feedback and support.

References

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6185-6194, 2023.

Xiaofeng Ren and Jitendra Malik. Learning a classification model for segmentation. In Proceedings ninth IEEE international conference on computer vision , pages 10-17. IEEE, 2003.

Max Wertheimer. Laws of organization in perceptual forms. A Source Book of Gestalt Psychology. , pages 71-88, 1938.

Roy Uziel, Meitar Ronen, and Oren Freifeld. Bayesian adaptive superpixel segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8470-8479, 2019.

Fengting Yang, Qian Sun, Hailin Jin, and Zihan Zhou. Superpixel segmentation with fully convolutional networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 13964-13973, 2020.

Aiping Zhang, Wenqi Ren, Yi Liu, and Xiaochun Cao. Lightweight image super-resolution with superpixel token interaction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12728-12737, 2023.

Varun Jampani, Deqing Sun, Ming-Yu Liu, Ming-Hsuan Yang, and Jan Kautz. Superpixel sampling networks. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 352-368, 2018.

Antoni Buades, Bartomeu Coll, and Jean-Michel Morel. Non-local means denoising. Image Processing On Line , 1:208-212, 2011.

Marc Lebrun. An analysis and implementation of the bm3d image denoising method. Image Processing On Line , 2:175-213, 2012.

Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng. Weighted nuclear norm minimization with application to image denoising. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2862-2869, 2014.

Ding Liu, Bihan Wen, Yuchen Fan, Chen Change Loy, and Thomas S Huang. Non-local recurrent network for image restoration. Advances in Neural Information Processing Systems , 31, 2018.

Tobias Plötz and Stefan Roth. Neural nearest neighbors networks. Advances in Neural Information Processing Systems , 31, 2018.

Ding Liu, Bihan Wen, Yuchen Fan, Chen Change Loy, and Thomas S Huang. Non-local recurrent network for image restoration. Advances in neural information processing systems , 31, 2018.

Hunsang Lee, Hyesong Choi, Kwanghoon Sohn, and Dongbo Min. Knn local attention for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2139-2149, 2022.

Gregory Vaksman, Michael Elad, and Peyman Milanfar. Lidia: Lightweight learned image denoising with instance adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 524-525, 2020.

Diego Valsesia, Giulia Fracastoro, and Enrico Magli. Deep graph-convolutional image denoising. IEEE Transactions on Image Processing , 29:8226-8237, 2020.

Yiqun Mei, Yuchen Fan, Yuqian Zhou, Lichao Huang, Thomas S. Huang, and Honghui Shi. Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

Chong Mou, Jian Zhang, Xiaopeng Fan, Hangfan Liu, and Ronggang Wang. Cola-net: Collaborative attention network for image restoration. IEEE Transactions on Multimedia , 24: 1366-1377, 2021.

Chong Mou, Jian Zhang, and Zhuoyuan Wu. Dynamic attentive graph learning for image restoration. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4328-4337, 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1833-1844, 2021.

Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multiscale vision transformer for image classification. In Proceedings of the IEEE/CVF international conference on computer vision , pages 357-366, 2021.

Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Crosscovariance image transformers. Advances in neural information processing systems , 34:2001420027, 2021.

Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels. Technical report, 2010.

Olga Veksler, Yuri Boykov, and Paria Mehrani. Superpixels and supervoxels in an energy optimization framework. In Computer Vision-ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part V 11 , pages 211-224. Springer, 2010.

Greg Mori, Xiaofeng Ren, Alexei A Efros, and Jitendra Malik. Recovering human body configurations: Combining segmentation and recognition. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. , volume 2, pages II-II. IEEE, 2004.

Jian Sun, Wenfei Cao, Zongben Xu, and Jean Ponce. Learning a convolutional neural network for non-uniform motion blur removal. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 769-777, 2015.

Kun Li, Yanming Zhu, Jingyu Yang, and Jianmin Jiang. Video super-resolution using an adaptive superpixel-guided auto-regressive model. Pattern Recognition , 51:59-71, 2016.

Yiding Liu, Siyu Yang, Bin Li, Wengang Zhou, Jizheng Xu, Houqiang Li, and Yan Lu. Affinity derivation and graph merge for instance segmentation. In Proceedings of the European conference on computer vision (ECCV) , pages 686-703, 2018.

Sree Ramya S. P. Malladi, Sundaresh Ram, and Jeffrey J. Rodríguez. Image denoising using superpixel-based pca. IEEE Transactions on Multimedia , 23:2297-2309, 2021. doi: 10.1109/ TMM.2020.3009502.

Wei-Chih Tu, Ming-Yu Liu, Varun Jampani, Deqing Sun, Shao-Yi Chien, Ming-Hsuan Yang, and Jan Kautz. Learning superpixels with segmentation-aware affinity loss. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 568-576, 2018.

Moshe Eliasof, Nir Ben Zikri, and Eran Treister. Rethinking unsupervised neural superpixel segmentation. In 2022 IEEE International Conference on Image Processing (ICIP) , pages 3500-3504. IEEE, 2022.

Constantinos Goutis. Nonparametric estimation of a mixing density via the kernel method. Journal of the American Statistical Association , 92(440):1445-1450, 1997.

David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001 , volume 2, pages 416-423. IEEE, 2001.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems , 35: 26565-26577, 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 , pages 234-241. Springer, 2015.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Charles R. Harris, K. Jarrod Millman, Stéfan J van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature , 585:357-362, 2020. doi: 10.1038/s41586-020-2649-2.

Wes McKinney et al. Data structures for statistical computing in python. In Proceedings of the 9th Python in Science Conference , volume 445, pages 51-56. Austin, TX, 2010.

NVIDIA, Péter Vingelmann, and Frank H.P. Fitzek. Cuda, release: 10.2.89, 2020. URL https://developer.nvidia.com/cuda-toolkit .

Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. Lowcomplexity single-image super-resolution based on nonnegative neighbor embedding. 2012.

Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5197-5206, 2015.

Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications , 76:21811-21838, 2017.

Pierre Charbonnier, Laure Blanc-Feraud, Gilles Aubert, and Michel Barlaud. Two deterministic half-quadratic regularization algorithms for computed imaging. In Proceedings of 1st international conference on image processing , volume 2, pages 168-172. IEEE, 1994.

A Additional Method Details and Proofs

A.1 Soft Superpixel Attention is the Optimal Denoiser

The latent superpixel model assumes the following data-generating process for each image pixel,

Sample the image pixel given the superpixel assignment x | s ∼ p ( x | s ) (11)

We will construct a denoiser that depends on the superpixel probabilities rather than the superpixel assignment. Our derivation is inspired by [37]. Only the form of p ( s i | π ( i ) ) = Categorical ( s i ; π ( i ) ) is known, and p ( π ) will not impact our final answer. So we only need to estimate the unknown density, p ( x | s ) . To do this, we approximate p ( x | s ) from a sample ( x , s , π ) ,

where p ( s i | m ) = π ( m, s i ) . By re-weighting the empirical density with superpixel probabilities, rather than superpixel assignments, our estimate does not depend on the ambiguous superpixel labels. Notice if the superpixel probabilities are independent of the sample index, then the usual weighting is recovered, p ( m | s i ) = p ( m ) = 1 M . Next, the pixels are corrupted with Gaussian noise,

Let the denoiser be a function of the noise intensity and superpixel probabilities, D (˜ x i ; σ, π ) . The expected error of the denoiser is taken with respect to the density ˆ p σ (˜ x i | s i ) p ( s i | π ( i ) ) p ( π ) ,

Note p ( s i = s | π ( i ) ) → p ( s | π ( i ) ) for clarity. The expectation can be minimized by minimizing each integrand for fixed a fixed noisy pixel (˜ x ) and superpixel probability ( π ) . Since the integrand is a sum of convex functions, the problem is convex and the solution is the point where the gradient is equal to zero.

In general, the optimal denoiser is written as,

By construction, the probability of a particular sample's superpixel assignment is written p ( s | m ) = π ( m,s ) . When the samples are restricted to a neighborhood surrounding pixel i , one can write the denoiser as the following expression,

This is soft superpixel neighborhood attention when the samples are restricted to a neighborhood surrounding pixel i , the qkv-transforms are identity, and the attention scale is fixed, λ at = 1 2 σ 2 .

Presently, we highlight where our per-pixel superpixel probabilities are used in the proof. In Equation 19, the probability p ( s | π ) depends on the sampled superpixel probabilities π . When denoising image pixel index i , we have p ( s | π ) = π ( i,s ) . However, under the standard model with a shape superpixel probability prior, p ( s | π ) does not depend on the particular pixel index. This reduces Equation 20 to a superpixel re-weighting term that does not depend on superpixel information from pixel i , which would be strange. We did experimentally try out this alternative re-weighting scheme, and its denoising quality matches NA.

Finally, we discuss a conceptual rationale for sampling superpixel probabilities for each image pixel. The introduction (Sec 1) explains that superpixel assignment is an ambiguous classification task . There are many qualitatively 'good' superpixel assignments, and often superpixel boundaries can be modified without impacting the overall quality. This suggests superpixel probabilities are more important than superpixel assignments. However, working with p ( x | π ) is more challenging than working with p ( x | s ) . So in this paper, we still work with p ( x | s ) but design the denoiser to depend on the probabilities π to circumvent using the ambiguous superpixel assignments.

A.2 Expected Denoising Error for a Fixed Neighborhood

Section 4.3 states that SNA is the optimal denoiser, but in practice, samples are limited to a neighborhood surrounding a query pixel. To analyze the impact of this restriction, this subsection computes the expected error over the image pixels and noise for a fixed superpixel sample. For expedient analysis, the number of features is one ( F = 1 , x i → x i ). The image pixels conditioned on the superpixel information are modeled with a Gaussian distribution, x i | s i , µ , η ∼ N ( µ s , η 2 s ) . We will analyze the attention modules by studying a denoiser of the following form,

This denoiser corresponds to using a flat attention scale λ at = 0 and using only prior superpixel probabilities as attention weights w j . Analysis of attention weights, in general, is difficult, and this approach removes the challenge. The expected error is computed over the Gaussian-corrupted and noise-free data density conditioned by the superpixel information, p (˜ x | x ) p ( x | s , µ, η ) . The expected error is split into the classic bias-variance decomposition. See Section A.4 for the proof.

The NA module's flat weights achieve the minimum variance of 1 W σ 2 where W = |N ( · ) | is the window size. However, NA's bias depends entirely on the superpixel's neighbors and can be large if the neighborhood happens to contain pixels from superpixels with different means. For example, the bias will be large for pixels along the sharp orange edge in Figure 3 since their neighborhoods include the dissimilar blue region. Section A.3 shows that even when the attention scale is non-zero, NA cannot robustly reject dissimilar regions.

H-SNA has the opposite problem of NA. H-SNA's sharp weights achieve the minimum possible bias of zero. However, H-SNA's variance depends entirely on the size of the superpixel. In Figure 3, the snake-like superpixel reduces the number of included pixels, increasing the variance term of the error. This observation that thin and small superpixels are undesirable matches existing assumptions from superpixel literature [26, 5].

SNA's modulation of the superpixel weights allows it to be better than both NA and H-SNA. In Figure 3, SNA excludes the dissimilar blue region (unlike NA) and includes the similar orange regions (unlike H-SNA). Said another way, SNA excludes only dissimilar regions.

A.3 Exclude Dissimilar Regions Instead of Dissimilar Pixels

Section A.2 showed SNA yields a smaller theoretical error than NA because SNA can exclude dissimilar regions via superpixel probabilities. The previous analysis fixed the attention scale to zero ( λ at = 0) , and one may wonder if the attention scale might be used to exclude dissimilar regions as well. This subsection shows the attention scale excludes dissimilar pixels rather than dissimilar regions, which is less robust under noise. In this analysis, the qk-transforms are fixed to identity, the feature dimension is fixed to 1 , and the distances function is the negative squared difference: d ( y i , y j ) = -( y i -y j ) 2 .

We first show the attention scale excludes dissimilar pixels rather than regions. Weights of pixels within a similar region should be similar, w i,j ≈ w i,i , while weights of pixels within dissimilar regions should be near zero, w i,j ≈ 0 . The decay of dissimilar regions will only occur for NA when the scale increases ( λ at > 0) , but this growth also decays the weights associated with similar regions. Noting e λ at d ( y i ,y i ) = 1 , the ratio of two weights for NA is as follows:

Two noisy pixel values must be approximately equal for their attention weights to be approximately equal. In other words, excluding dissimilar regions with the attention weights also excludes nonidentical noisy pixels. This strict condition limits NA's ability to exclude dissimilar regions.

Compare this with SNA, which we argue excludes regions more robustly. Say all but a single region should be excluded, so λ sp can be large. The inclusion of the single (useful) region depends on the most likely estimated superpixel matching the latent probabilities, s ∗ i = ˆ s i . Using Section 3.1's formulation, the argmax terms match when the following condition is met,

̸

In other words, SNA properly rejects dissimilar regions when a noisy pixel is more similar to its true mean than another superpixel mean. Comparing two noisy differences is more robust than comparing two noisy pixels. So we claim SNA rejects dissimilar regions more robustly than NA.

A.4 The Estimated Expected Denoising Error

This subsection presents a more detailed version of Claim 3 from Section A.2 and its proof.

Claim 3 The expected error of the denoiser D over the joint density p (˜ x , x | s , µ , η )

The following proves the above claim. The first step is to expand the square,

Each expectation is evaluated separately,

Then all terms are put together and rearranged,

Using E [ X 2 ] = Var[ X ] + E [ X ] 2 , the bias-variance terms can be identified. Since the E [ X ] 2 term is simple to compute and exactly matches the right-hand side, we can label those terms as the bias and assign the remaining terms to the variance.

0=10

=20

=30

33.2

29.7

27.9

32.4

28.4

26.3

B

PSNR

31.7

27.2

24.8

Method

P

31.0

SNA-Deep

25.9

23.2

NA-Fixed

H-SNA

30.2

24.6

X

21.7

X

32

52

92

152

212

32

52

92

152

212

32

52

92

152

212

Neighborhood Size(pixels)

Neighborhood Size(pixels)

NeighborhoodSize(pixels)

0=10

0=20

=30

32.7

29.8

28.5

32.2

29.1

27.8

B

Method

SNA-SLIC

PSNR

31.8

SNA-Deep

28.5

27.1

NA-Learn

31.3

27.8

26.4

30.9

X

27.1

X

25.7

5

40

70

150

5

40

70

150

5

40

70

150

Aux Net Size (103 Params)

Aux NetSize(103Params)

Aux Net Size (103 Params)

0=10

=20

=30

32.24

Method

29.1

27.6

SNA-SLIC

SNA-Deep

B

32.15

29.0

27.5

PSNR

32.07

28.8

27.3

31.98

28.7

27.1

4

6

7

10

12

17

4

6

7

10

12

17

4

6

7

10

12

17

Num. of Superpixels (102)

Num. of Superpixels (102)

Num.of Superpixels(102)

B Additional Experiments

B.1 Ablation Experiments

This subsection explores the impact of hyperparameters related to the proposed SNA. This includes the neighborhood window size (Fig 9), the size of the auxiliary network (Fig 10), and the number of superpixels (Fig 11). All networks are trained and tested according to Section 5.2. Both the size of the auxiliary network and the neighborhood window size have a significant impact on the denoiser quality (over 1 dB). The number of superpixels has a smaller overall impact (less than 0 . 5 dB).

B.2 Additional Denoising Results

SNA

NA

Noisy

Clean

Learn入at

Fix 入at

Learn入at

23.39

32.38

31.71

29.11

29.33

22.22

29.08

29.57

26.66

27.91

22.35

29.03

29.77

27.08

27.44

22.26

30.48

30.37

29.03

28.08

23.08

28.13

28.81

23.02

27.55

22.93

24.77

26.38

24.17

23.25

22.15

29.44

29.46

28.12

27.02

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: This paper presents soft superpixel neighborhood attention (SNA), which extends standard neighborhood attention to account for deformable boundaries within natural images. Sections 4.3 and A.1 claim and prove SNA is the optimal denoiser under the latent superpixel model. Section 5.2 empirically demonstrates SNA outperforms alternative methods on Gaussian denoising, validating our theoretical finding. Section 5.3 compares the superpixels learned from the denoising task to superpixels learned with explicit supervision.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Table 1 Section 5.2 reports the proposed SNA module requires more computation than the existing baseline method (NA). We note the implementation of SNA has not been optimized, which can dramatically reduce wall-clock time. The primary optimization is to more efficiently read from global CUDA memory, which may have a dramatic impact on wall-clock time. Section 4.3 states SNA is the optimal denoiser, but assumes the corrupting noise is Gaussian.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Section 4.3 includes the main theoretical contribution of this paper. All assumptions are stated in the statement of the proof. Intuition for an important assumption is described in Section 4.2 and visualized in Figure 4. Section A.1 includes the formal proof of the statement.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section 4.1 writes the proposed attention module using standard notation. The experiment set-up described in Section 5.1 explains the important parameters used to run the denoising and superpixel-loss experiments. The denoising experiment uses the random seed 123 across all test samples. The code is provided in the supplemental material and will be made publicly available upon acceptance.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is provided in the supplemental material and will be made publicly available upon acceptance. The code is implemented in Python using Pytorch, Numpy, Pandas, and CUDA and run using two NVIDIA Titan RTX GPUs and one RTX 3090 Ti GPU [40-43]. The code includes a readme, and requirements files to setup the environment. Testing datasets are Set5 [44], BSD100 [36], Urban100 [45], and Manga109 [46].

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experiment set-up described in Section 5.1 explains the important parameters used to run the denoising and superpixel-loss experiments. We train each network for 800 epochs using a batch size of 2 on the BSD500 dataset [36] using a learning rate of 2 · 10 -4 with a decay factor of 1 / 2 at epochs 300 and 600. These hyperparameters were chosen using an informal grid search. The code is provided in the supplemental material and will be made publicly available upon acceptance.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars are not usually reported for the benchmarks used in this paper, and we believe including them will add unnecessary clutter to the report. If requested by the reviewers, the 95% confidence interval of the average PSNR/SSIM can be added to the experiments. However, generating error bars to assess the impact of training requires rerunning all of the experiments several times, which significantly increases the computational cost of this paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Section 5.1 states the workstation used to run the experiments includes two NVIDIA Titan RTX GPUs and one GeForce RTX 3090 Ti GPU. Each experiment takes about 2 hours to run, and all experiments together take less than 72 hours to run. This paper did require additional computing to determine if using the simple deep learning network was necessary, given the computing budget. This entailed training several larger deep networks and iteratively pruning network layers to reduce the wall-clock training time. Ultimately, using a very simple network enabled a clear, simple experiment to demonstrate this paper's major claims.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper presents a module for deep neural networks and applies the module to single-image denoising. While this module may improve generic deep neural networks, this specific paper is not tied to any direct application.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper proposes a module for denoising and is not at risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The code is implemented in Python (PSF) using Pytorch (BSD3), Numpy (BSD), Pandas (BSD3), and CUDA (NVIDIA Software License) [40-43]. The testing datasets are Set5 [44], BSD100 [36], Urban100 [45], and Manga109 [46]. These datasets are properly referenced with their supporting paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: This paper proposes code for a new attention module, and the code is provided in the zip file. Within the zipped code file, the new asset is located within the superpixel_paper/lib/superpixel_paper/ssna/ directory. Two important files within the directory are superpixel_paper/lib/superpixel_paper/ssna/ssna.py and super-pixel_paper/lib/superpixel_paper/ssna/attn_reweight.py. The code will be released under the BSD-3-Clause license.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper proposes soft superpixel neighborhood attention, with the motivation that attention is more efficient when it is local, and superpixels are better for local attention as patches, and furthermore, soft superpixels are more robust to errors than hard superpixels.

STRENGTHS:
The idea of soft superpixel neighborhood attention makes sense. 

Illustrations in the paper are good and appropriate.

WEAKNESSES:
The technical novelty in the paper is not major.

The application (denoising) is not very appealing in the sense that it is not such an important application in the computer vision. It would be best to evaluate the proposed approach on some applications that are more widely used, such as semantic segmentation.

There is a theoretical contribution of optimality of the proposed denoiser is proven, but it assumes the proposed latent superpixel model, and it is not clear how true of the reality this proposed model is.

QUESTIONS:
see above

LIMITATIONS:
Limitations are addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-06-25

comments: SUMMARY:
The paper proposes a new attention mechanism based on Soft Superpixel Attention. The main idea is to use superpixel segmentations directly in the attention module. The paper proposes rigorous proof that the proposed mechanism is the optimal denoiser. Results show an improved denoising performance.

STRENGTHS:
* The paper presents an elegant and novel idea to use superpixels in the attention module.
* It is well-written with clear explanations. I appreciate that parts that are less important for paper understanding are moved to the supplementary materials (e.g. proofs).
* The experiments are sufficient with an improved performance over the state-of-the-art.

WEAKNESSES:
* The performance improvement is relatively small.
* The paper would be much stronger if it could show how this attention module could be used in other tasks than only on denoising. For instance, deblurring, object detection, tracking, etc.

QUESTIONS:
How difficult is it to incorporate the proposed attention mechanism in other downstream tasks?

LIMITATIONS:
Yes, this is discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-08

comments: SUMMARY:
This paper proposes a soft superpixel neighborhood attention (SNA). It proves that SNA is the optimum denoiser under Gaussian noise. Experiments show that SNA outperforms other local attention modules for the image denoising task.

STRENGTHS:
- This is an interesting theoretical study, backed up with experiments.
- It is refreshing to see some theoretical work submitted to Neurips
- Using non-rigid boundaries instead of square neighborhood makes sense. 
- Results are promising

WEAKNESSES:
- Experiments are limited. The network that has been tested is written in Eq. (10).
- The current implementation is slow. Some effort is needed to program an efficient version, but this is left for future work.

QUESTIONS:
I am not sure that I understand the training process. It is claimed that the network is trained on BSD500, but do you train by adding noise to the image, and targeting the network to produce the clean, original image?

LIMITATIONS:
The hypothesis for the theory are clearly stated, and the limitation are acknowledged.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an attention module in which the dot product weights are modified with\nsuperpixel probabilities, named Superpixel Neighborhood Attention (SNA). By doing so, the\noptimization process is arguably made easier by letting attention avoid learning spurious\ninteractions, which prior work into the attention module has found to be somewhat common. \nOne key difference to prior works in superpixels is modelling superpixels as probabilities instead\nof naive binary interactions, which in theory allows a flexible interpolation between standard\nattention and superpixel attention. The proposed module is shown to improve the baseline approach\nwithout superpixels by around 1 to 3 dB PSNR, and qualitative results support this claim as well.\n\nSTRENGTHS:\n1. Unique approach for injecting superpixel information directly into an attention operator. From \nmy point of view, this is among the most significant contributions of the paper, which would enable\nimproved achievable performance with attention in the context of larger resolution problems, of\nwhich denoising is one. I think the impact of this approach extends beyond denoising and super\nresolution applications.\n\n2. Significant improvement from the baseline approach, neighborhood attention. By allowing\nirrelevant local interactions to be avoided, SNA provides considerably better denoising quality than\nthe standard neighborhood attention.\n\nWEAKNESSES:\n1. From my point of view, the most considerable weakness is a lack of discussion on the\ncomputational complexity of the proposed approach. Note that I am not referring to the reported\nperformance gap, but generally to the fact that it is unclear from reading the paper how complex the\nSNA attention algorithm is, and whether or not it can scale beyond the scope of the experiments done\nin this paper. While metrics such as time and space complexity, and similar metrics such as FLOPs\ndon't necessarily translate into actual efficiency, they do provide a rough idea of whether or not\ntwo operations are at least comparable in terms of resource requirement in theory. Given that the\nbaseline, neighborhood attention, is seemingly so much more efficient in runtime and memory usage,\nand the fact that it has been performance optimized to some extent over time, and that these are\nboth operations in complex deep neural networks implemented primarily by a deep learning framework\nwith a different standard of implementation, I think the reported numbers are simply not that\ninformative. The disclosure and clarity is certainly appreciated, but stating that the proposed\napproach's performance levels are \"likely overly pessimistic\" is not informative, and could be\neasily reworded by providing some evidence that there isn't such a significant difference in\nperformance levels in theory.\n\n2. Lack of comparison to other relevant approaches. I think limiting the use of dot product\nattention to just one local pattern, namely neighborhood attention, is worth a discussion. Is there\na reason why self attention itself is not considered here, given that the superpixel probabilities\neffectively serve as an implicit attention mask. It may also be true that one can map superpixel\nprobabilities directly to attention weights and define attention differently. I think while the\npaper is mostly complete as is, this is something that merits at least a paragraph.\n\n3. The citation for neighborhood attention is incorrect; the reference (\"Neighbor2neighbor\") does\nnot mention neighborhood attention. The correct reference is:\n\n>Hassani, A., Walton, S., Li, J., Li, S. and Shi, H., 2023. Neighborhood attention transformer. \n>In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6185-6194).\n\nQUESTIONS:\n1. Is there any metric through which SNA's cost or resource requirement can be compared against\nstandard neighborhood attention with more certainty? The seemingly large performance gap in terms of\nactual runtime and memory footprint, while necessary, is unsurprising, and only really reveals that\nas part of future efforts, SNA could use performance optimization. However, it is completely unclear\nhow far that optimization could go. My suggestion is at the very least providing some other proxy\nfor that, be it in the form of time and space complexity, or any other relevant analysis of the\nsteps required by the algorithm and how easy / difficult they will be to parallelize and performance\noptimize.\n\n2. Can you elaborate Eq (1)? Softmax is an operation over a set or vector, and I would assume it\nwould be over N values per pixel, assuming each pixel has N superpixels.\n\n3. This is just out of curiosity, but as of recently the neighborhood attention package does support\nnon-square kernels. Does that change in assumptions with regard to neighborhood attention change\nanything about the significance of SNA over NA, or is that just a minor detail?\n\n4. How are the efficiency performance metrics (runtime and memory usage) evaluated? Were standard\npractices followed? (i.e. locking power and frequency, benchmarking without external factors such as I/O,\ncommunication, framework-level optimizations such as caching allocators and memory planning,\nmeasuring metrics iteratively and reporting an average, and the like)\n\n5. Do either of SNA / H-SNA use the original neighborhood attention implementation, or is the\nimplementation done in this paper completely specific to superpixels? I'm trying to understand\nwhether SNA or its future application could also use additional performance optimizations done to\nneighborhood attention, or whether it would require its own independent development and in turn face\nsimilar issues in terms of extensibility as any other such approach.\n\nLIMITATIONS:\nWhile the reported performance gap is a limitation of the proposed approach at the moment, I think\nit is very insignificant when taking into account the rest of the contributions of the paper.\n\nHowever, I cannot be sure of that without evidence that would suggest the performance gap is simply\ndue to a lack of performance optimization. It could very well be the case that SNA in its current\nform cannot scale easily beyond the scope of this paper, which would be in my opinion a far greater\nlimitation.\n\nThis is therefore either a limitation (meaning it is simply not possible or extremely difficult to\nanalyze SNA's computational cost and compare against standard neighborhood attention) or a weakness\n(it was just not included in the paper.)", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-14"}