Soft Superpixel Neighborhood Attention

Authors: kent  gauen
stanley  chan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 21:40:48

Abstract: Images contain objects with deformable boundaries, such as the contours of a human face, yet attention operators act on square windows. This mixes features from perceptually unrelated regions, which can degrade the quality of a denoiser. One can exclude pixels using an estimate of perceptual groupings, such as superpixels, but the naive use of superpixels can be theoretically and empirically worse than standard attention. Using superpixel probabilities rather than superpixel assignments, this paper proposes soft superpixel neighborhood attention (SNA), which interpolates between the existing neighborhood attention and the naive superpixel neighborhood attention. This paper presents theoretical results showing SNA is the optimal denoiser under a latent superpixel model. SNA outperforms alternative local attention modules on image denoising, and we compare the superpixels learned from denoising with those learned with supervision.

Main paper: 
 introduction: 
 The attention mechanism is attributed to meaningful benchmark improvements in deep neural networks [1] . While the initial attention mechanism acted globally, recent methods operate on local neighborhoods, which requires less computation and reduces the chance of learning spurious correlations [2] . Neighborhood attention (NA) transforms each pixel using its surrounding square neighborhood of pixels. In reality, this square neighborhood is merely an implementation convenience. Natural images contain objects with non-rigid boundaries, such as the contours of a human face or the outlines of text. To account for these deformations, this paper proposes a new local attention module that re-weights the attention map according to low-level perceptual groups, or "superpixels" [3] , named soft superpixel neighborhood attention (SNA).Theory from vision science posits that humans view scenes according to perceptual groups (e.g. Gestalt principles) [4] . Superpixels are one such formulation of perceptual groupings, and they create a deformable, boundary-preserving segmentation of an image [3] . Since deep learning models can learn spurious correlations, operations that utilize perceptual groups should principally help a network learn desirable correlations. Despite the deformable neighborhood shapes, applying attention with a superpixel estimate is straightforward. However, this naive approach is theoretically worse than NA for some superpixel shapes and sensitive to errors in the superpixel estimate. To address these issues, we propose thinking beyond using a single superpixel estimate.While recent works use the maximum likelihood (ML) superpixel estimate [6, 7] , many superpixel formations are equally likely. Figure 1 depicts three samples from a (slightly modified) recent BASS superpixel model [5] . Many of the boundaries seem arbitrary and change among the samples, while some remain fixed. No single superpixel segmentation is the absolute "best". This is not a deficiency of the particular method to estimate superpixels, but rather a drawback of using a point estimate.Figure 1 : The Ambiguity of Superpixels. This figure compares three superpixel estimates from a recent method named BASS. [5] 2 While all three samples achieve a similar segmentation quality, some regions are different, and some are almost identical. Since no single segmentation is the "best", this suggests that superpixel assignments are not as important as superpixel probabilities.This paper presents a theoretically grounded approach to account for the ambiguity of superpixel assignments within an attention module. We consider this ambiguity in our derivation by re-weighting the empirical data distribution with superpixel probabilities. While the standard superpixel models assume a fixed prior probability, this paper supposes each pixel has a separate prior superpixel probability. Under this new model, we find the optimal denoiser is the proposed SNA module.Contributions. In summary, this paper presents a rigorous way to incorporate superpixels into an attention module. By modeling image pixels with distinct superpixel probabilities, we find the soft superpixel neighborhood attention (SNA) module is the optimal denoiser. We empirically show the SNA module outperforms NA within a single-layer denoising network by 1 -2 dB PSNR. We compare the superpixels learned for denoising with superpixels learned using supervised training [8] .Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: This paper presents soft superpixel neighborhood attention (SNA), which extends standard neighborhood attention to account for deformable boundaries within natural images. Sections 4.3 and A.1 claim and prove SNA is the optimal denoiser under the latent superpixel model. Section 5.2 empirically demonstrates SNA outperforms alternative methods on Gaussian denoising, validating our theoretical finding. Section 5.3 compares the superpixels learned from the denoising task to superpixels learned with explicit supervision.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related works: 
 Non-Local Methods. Classical image restoration methods transform a pixel using its neighborhood based on pairwise similarities [9] [10] [11] . Some early deep learning models incorporate non-local modules with deep learning models for image restoration [12] [13] [14] [15] [16] [17] . Once attention was introduced to the computer vision community, later methods drew analogies to between attention and non-local methods [18] [19] [20] .Attention. While attention modules followed the same form as non-local methods, they were originally presented as a disconnected idea [1, 21] . Attention's primary issue was originally computation, and subsequent research efforts proposed efficient alternatives [22] [23] [24] [25] 2] . These alternatives often compute attention across a smaller region, and use a deep network's depth to disperse information globally. Neighborhood attention (NA) is implemented with an efficient, custom CUDA kernel to search a square window and is among the fastest methods [2] . The square window is theoretically more desirable than asymmetrical alternatives, such as SWIN [22] . Superpixels. Superpixels embody the concept of perceptual groupings, which suggests that humans first group smaller objects together and then aggregate this information to understand the whole image [3] . There are many methods to model these superpixels [26, 27] . SLIC is a state-of-the-art, classical superpixel method and is explained in Section 3.1 [26] . Superpixels have been applied extensively outside of deep learning literature [28] [29] [30] [31] [32] . Superpixels have not been employed as widely since the advent of deep learning, but recent methods have started incorporating them [6, 7] . A recent method, referred to as the superpixel sampling network (SSN), presents a supervised learning method to sample using segmentation labels [8] . The loss function of SSN is related to the idea of superpixel pooling, which we use to evaluate superpixel probabilities in Section 5.3 [33, 34] .Non-Parametric Density Estimation. The derivation of the optimal denoiser using latent superpixels closely resembles the problem of mixture-based non-parametric density estimation [35] . This paper does not address important statistical issues, such as identifiability. Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Table 1 Section 5.2 reports the proposed SNA module requires more computation than the existing baseline method (NA). We note the implementation of SNA has not been optimized, which can dramatically reduce wall-clock time. The primary optimization is to more efficiently read from global CUDA memory, which may have a dramatic impact on wall-clock time. Section 4.3 states SNA is the optimal denoiser, but assumes the corrupting noise is Gaussian.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
slic superpixels: 
 In this paper, an image (x) has height (H), width (W), and features (F ) with shape HW × F . Conceptually, superpixels are evenly spaced across the image according to the stride denoted S sp . The superpixel stride (S sp ) determines the number of superpixels, S = HW/S 2 sp (see Fig 2) . SLIC superpixels consist of superpixel means ( μ with shape S × F ), intra-superpixel precision ( η = [ ηapp ηshape ] with shape S × 2), the probability each pixel belongs to each superpixel ( π with shape HW × S), and the superpixel assigned to each pixel (ŝ with shape HW). SLIC estimates these quantities using Lloyd's algorithm (similar to k-means) [26] . Following recent works, each pixel is connected to at most nine superpixels [8] .Presently, we describe how SLIC superpixel probabilities are computed. Let the distances between pixels and superpixel means be written as a matrix D of shape HW × S and letD (i,s) = ∥x i -µ s ∥ 2 2if the two are connected and infinity otherwise. A difference between the indices is writtenD (i,s) = ∥[i x i y ] -[s x s y ]∥ 22 . Superpixel probabilities are computed as,EQUATIONwhere ηi,app , ηi,shape > 0 enforce consistency across appearance (app) and shape. The estimated superpixel probabilities are the conditional probability pixel i is assigned to superpixel s, p(s i = s|x i ) = π(i,s) . A superpixel segmentation (ŝ) is formed by assigning each pixel's cluster label to their most similar superpixel; ŝi = arg max s π(i,s) . These quantities are estimates (note the hats) of latent superpixels, which is a modeling assumption discussed in Section 4.2.The attention module proposed in this paper will learn superpixel probabilities, but standard superpixel evaluation only considers a point estimate. To augment the standard superpixel benchmarks, we follow recent works and compare superpixel probabilities with superpixel pooling [8, 5, 34] . A matrix z has size HW × C for C segmentation classes, and superpixel probabilities ( π) have size HW × S. Then, the probabilities are re-normalized, π(i,s) = π(i,s) / HW j=1 π(j,s) . Finally, the superpixel pooling operation is written,EQUATIONIn this paper, the vector z is either an image (C = F is the number of channels) or a segmentation label. A recent method, named the Superpixel Sampling Network (SSN), proposes learning taskspecific superpixels with this superpixel pooling loss [8] . Their loss function trains a network to estimate superpixel probabilities by comparing the original and pooled vectors. In Section 5.3, we compare the quality of superpixels learned within an SNA-denoising network and those learned using the SSN loss on the BSD500 dataset [36] .Figure 3 : Superpixel Neighborhood Attention. The yellow region represents the attention window and the red contours are a superpixel boundary. NA considers all pixels, mixing the dissimilar orange and blue pixels. H-SNA considers only pixels within its own superpixel, which is too few pixels for denoising. SNA excludes the dissimilar blue pixels but retains the similar orange pixels.Let a noisy input image be denoted x with shape HW × F . The queries (q), keys (k), and values (v) project the image x from dimension F to D, written asq = xW q , k = xW k , v = xW v .Neighborhood attention (NA) computes the output of the attention operator using pixels within a square neighborhood around pixel i, denoted N (i) [2] . A standard neighborhood of size 7 × 7 is among the fastest available attention methods. The attention scale controls the shape of the attention weights, λ at ≥ 0. The output and attention weights are computed below,EQUATIONAs an additional baseline, this paper augments the standard NA by learning the attention parameter for each pixel. Specifically, each pixel's attention scale is the output of a deep network, λϕ,Deep (x). Further details are described in Section 4.5.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: Section 4.3 includes the main theoretical contribution of this paper. All assumptions are stated in the statement of the proof. Intuition for an important assumption is described in Section 4.2 and visualized in Figure 4 . Section A.1 includes the formal proof of the statement. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
superpixel neighborhood attention: 
 Hard Superpixel Neighborhood Attention (H-SNA). Perceptual groupings, such as superpixels, segment pixels into a boundary-preserving partition of the image. These boundaries account for sharp changes in pixel intensity, so using these segmentations as masks can remove perceptually unrelated information. Supplemental Section A.2 formalizes this intuition. H-SNA is a naive implementation of this concept. For a fixed query pixel (i), a neighboring pixel (j) is masked if its estimated superpixel does not match the query's superpixel, ŝi ̸ = ŝj . The function is written as,EQUATIONSoft Superpixel Neighborhood Attention (SNA). There are two problems with H-SNA. First, H-SNA is unstable when the number of pixels sharing the same superpixel label within the local neighborhood is small. Examples include snake-like tendrils and superpixels that consist of small, disconnected regions (see Section A.2). The second problem with H-SNA is its high sensitivity to improper superpixel estimation. Erroneous ML estimates of superpixels have a dramatic effect on quality. To mitigate both issues, we propose soft superpixel neighborhood attention (SNA), which uses superpixel probabilities instead of superpixel assignments,f (i) SNA (x; π) = j∈N (i) w i,j v j , w i,j = exp (λ at d(q i , k j )) S s=1π(i,s) π(j,s)EQUATIONFigure 4 : The Latent Superpixel Model. The latent superpixel model assumes superpixel probabilities are sampled for each image pixel. This figure illustrates the data-generating process. The leftmost image allows the reader to visually compare the similarities among superpixels by representing each pixel by its most likely superpixel means. Informally, this looks like a "low-resolution image". The superpixels and image pixels are sampled as usual.Importantly, the sum over all superpixels contains only 9 non-zero terms (see Section 3.1). The shape of the superpixel probabilities allows SNA to interpolate between NA and H-SNA. To be specific, the superpixel probabilities may be flat to match NA, π(i,s) ≈ 1 9 , or the superpixel may be sharp to match H-SNA, π(i,s) ≈ 1[s = s * ]. This control over the shape of the superpixel probabilities allows SNA to be better than both H-SNA and NA on average.Properly incorporating superpixels into attention requires rethinking the relationship between superpixels and images. While superpixels are usually part of the generative model for images, the superpixel model is not used beyond estimating superpixel assignments. In contrast, this paper uses the data-generating process itself to derive the form of our proposed attention module. Our process is slightly different from standard literature. Ordinarily, all pixels share a single prior superpixel probability. In this paper, the superpixel probabilities are sampled for each image pixel.For each image pixel, our generative model samples (1) superpixel probabilities π ∼ p(π), (2) superpixel assignments s i ∼ p(s i |π (i) ), and (3) pixel values x|s ∼ p(x|s). Figure 4 illustrates this proposed latent variable model. Note that SLIC estimated these quantities (recall the hats: ŝ, π), but assumes a different data generating model. While the distribution of pixels given the superpixel assignments is unknown, the new model is useful for theoretical analysis. We use it to derive the form of our proposed attention module (Sec 4.3), and to compare the theoretical error between SNA, NA, and H-SNA (Sec A.2).The optimal denoiser is the function that minimizes the mean-squared error between the denoised pixel and the original pixel value. Following similar derivations [37] , minimizing this expectation with respect to an unknown denoiser (D) is possible because of our chosen model for the unknown data density. Say we have a sample, x, s, π ∼ p(x, s, π). Then we approximate p(x i |s i ),EQUATIONsince p(s i ) = 1 9 , and p(m) = 1 M . By re-weighting with superpixel probabilities, rather than superpixel assignments, our estimate does not depend on ambiguous superpixel labels. The expected loss of a denoiser is computed over the factorized joint density of the Gaussian-corrupted pixels, noisefree pixels, superpixel assignments, and superpixel probabilities: p( x i |x i )p(x i |s i )p(s i |π (i) )p(π).The optimal denoiser to the following optimization problem is soft superpixel neighborhood attention (SNA) when the qkv-transforms are identity, the attention scale is fixed to λ at = 1 2σ 2 , and the samples used to estimate the data density come from a neighborhood surrounding pixel i,EQUATIONFigure 6 : Estimating Superpixel Probabilities. The superpixel probabilities are learned during training using one of these two methods. The left method uses a shallow UNet followed by a softmax layer with nine channels (g ϕ,Deep ). The right method estimates hyperparameters to be used within SLIC iterations (g ϕ,SLIC ).See Supplemental Section A.1 for the proof. In practice, the error of a denoiser depends on the sampled data, and in this paper, the data is limited to a neighborhood surrounding the query point. Section A.2 analyzes this limited neighborhood. SLIC [26] and SNN [8] connect each pixel to (at most) nine superpixels. Therefore, some pixels within a neighborhood may only connect to a subset of superpixels that connect to the query index. This sharp cut-off in connectivity creates artifacts within the attention map since the number of non-zero superpixel probabilities for some neighbors is fewer than nine. These artifacts are due to our choice of superpixel algorithms [26, 8] rather than a limitation of our attention module. To correct these artifacts for the attention map associated with pixel i, we re-normalize the adjacent superpixel probabilities to one; π (j,s) → π (j,s) / s ′ ∈C(i) π (j,s ′ ) , where C(i) is the set of superpixels connected to pixel index i. Figure 5 illustrates the attention maps with and without the normalization step.SNA re-weights the attention map using superpixel probabilities, which are estimated from noisy images, π ≈ π = g • (x). This paper forgoes considering theoretical concerns (such as identifiability) and uses ad-hoc estimates of these probabilities with two deep networks, depicted in Figure 6 .The first method estimates the marginal probabilities directly from a deep network, π = g ϕ,Deep (x).The second method estimates the superpixel parameters controlling the balance between appearance and shape with a deep network, which is then fed into differentiable SLIC iterations [8] (Section 3.1): π = g ϕ,SLIC (x, η) where [ ηapp ηshape ] = gϕ,SLIC (x). This model is abbreviated as π = g ϕ,SLIC (x).The number of SLIC iterations is fixed to 5, following a recent paper [7] . These networks are two-layer UNets [38] .Using a network similar to the ones used to estimate superpixel probabilities, all attention modules are augmented to serve as another baseline. The augmentation replaces the fixed attention scale with learnable attention scales for each pixel, λ at = g ϕ,Attn (x). The network architecture is identical to g ϕ,Deep except for the final layer, which outputs one channel rather than nine. This baseline empirically demonstrates the theoretical argument from Supplemental Section A.3; even when modulating the attention scale, NA cannot robustly reject dissimilar pixels.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Section 4.1 writes the proposed attention module using standard notation. The experiment set-up described in Section 5.1 explains the important parameters used to run the denoising and superpixel-loss experiments. The denoising experiment uses the random seed 123 across all test samples. The code is provided in the supplemental material and will be made publicly available upon acceptance. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
experiments: 
 This section demonstrates the impressive benefit of superpixel neighborhood attention compared to standard neighborhood attention. To verify whether the improvement is due to the proposed method, we compare several variations of both attention methods. Section 5.2 compares different attention modules within a simple network architecture on Gaussian denoising, which empirically verifies the theoretical findings in Sections 4.3 and A.2. Section 5.3 compares the superpixel probabilities learned from the denoising loss function with superpixels learned through supervised training. Supplemental Section B.1 includes ablation experiments.Our experiments use a simple denoising network with larger auxiliary networks to highlight the role of the learned superpixel probabilities and attention scale parameters. The denoising network contains only projection layers and a single attention module. To keep the extracted features simple, the only interaction between neighboring pixels is through the attention layer. The project matrices project the input RGB image from three to six dimensions dimension. The full network is written,EQUATIONThe primary network parameters are denoted θ = [W 0 , W 1 ], and g ϕ (•) is the auxiliary deep network from Section 4.5. The primary network consists of only 200 parameters (θ), and the auxiliary networks contain about 4.4k or 8.8k parameters (ϕ). We train each network for 800 epochs using a batch size of 2 on the BSD500 dataset [36] using a learning rate of 2 • 10 -4 with a decay factor of 1/2 at epochs 300 and 600. The network is optimized with Adam [39] . The code is implemented in Python using Pytorch, Numpy, Pandas, and CUDA and run using two NVIDIA Titan RTX GPUs and one RTX 3090 Ti GPU [40] [41] [42] [43] . Testing datasets are Set5 [44] , BSD100 [36] , Urban100 [45] , and Manga109 [46] .This subsection empirically verifies the utility of SNA on Gaussian denoising. The Charbonnier loss is used for denoiser training, L Deno = ∥ ŷDeno -y∥ 2 + ε 2 (ε = 10 -3 ) [47] . By default, the window size is 15 × 15, the fixed SLIC precision is ten, and the fixed attention scale is one.Table 1 presents the quantitative results. The SNA module achieves far better denoising quality than NA, even when NA can learn its attention scale. As an extreme example, the PSNR at σ = 30 of the left-most SNA column is about 2.8 dB better than standard NA (2nd NA column). If comparison is restricted to models with the same number of parameters, the PSNR at σ = 30 of the the fourth SNA column is about 1.8 dB better than the augmented NA (1st NA column). For a fixed budget of network parameters, SNA yields a higher quality denoiser than NA.Figure 7 shows SNA is qualitatively superior to NA. The first row shows that SNA produces less grainy images than NA. The bottom two rows show that NA mixes perceptually unrelated information.In the second row, NA outputs a red-orange semi-transparent mist surrounding the orange-red rock. In the third row of the NA results, the white zebra stripes are shaded darker than the clean image when the attention scale is learned, and contain white speckles if fixed. See Section B.2 for more examples.Learning the attention scale improves standard NA and SNA+g ϕ,Deep at higher noise intensities. However, it degrades model quality when superpixels are estimated with SLIC iterations (SNA+g ϕ,SLIC ). Perhaps this quality drop is simply due to network architecture and/or the flow of information. Since the network that learns the attention parameter is only given a noisy image, the simple model struggles to coordinate with the denoising network, which uses multiple SLIC iterations. SLIC iterations explicitly compute pairwise distances, which cannot be learned with the shallow model.A limitation of the proposed SNA module is the additional computation compared to NA. SNA amounts to re-weighting the attention map produced by a neighborhood search, so SNA is necessarily more compute-intensive than NA. The bottom four rows of Table 1 show SNA is 15 -22 times slower than NA. While this increase is significant, we believe the numbers reported in this paper are overly pessimistic. Our implementation of SNA has not been optimized, and future development can dramatically reduce the wall-clock time. One way to reduce wall-clock time is to read more efficiently from global CUDA memory [43] . Anecdotally, the naive implementations of NA are 500% to 1500% slower than their recently optimized alternatives.This subsection investigates how the superpixels learned from denoising compare with superpixels learned from explicit supervised superpixel training [8] . We observe a collaborative relationship between superpixel probabilities for denoising and superpixel pooling. We observe an adversarial relationship between the superpixels for denoising and boundary adherence. Generally, learning superpixels that improve superpixel benchmarks decreases their utility for denoising. To compare the different training regimes for learning superpixel probabilities, an SNA network is fully trained with a combination of two loss functions. One loss is the denoising loss (L Deno ).The second loss is the SSN loss function, which computes the difference between a target vector and its superpixel-pooled transformation (see Section 3.2 for super-pixel pooling) [8] . Specifically, L SSN = L( ẑsp-pool , z) where L is the cross-entropy loss for a segmentation label and the meansquared-error for image pixels. The superpixel stride for all methods is fixed to S sp = 14. Each network estimates superpixels using SLIC iterations, and the attention scale is fixed to one (if applicable). Evaluation is computed on the test set of BSD500 [36] .The Achievable Segmentation Accuracy (ASA) and Boundary Recall (BR) scores evaluate the quality of the maximum likelihood superpixel assignment. The ASA score measures the upper bound on the achievable accuracy of any segmentation method, and the BR measures the boundary alignment between the superpixels and a segmentation. The superpixels are processed with a connected components algorithm before computing ASA and BR scores. To qualitatively compare the learned superpixel probabilities, we follow recent literature and compare the superpixel pooled images [5] .Images are compared against their superpixel pooled images using PSNR and SSIM.Table 2 evaluates the denoising quality, Table 3 quantitatively evaluates the superpixel quality, and Figure 8 qualitatively evaluates the superpixel pooling quality. We observe a harmony between the denoising loss and pixel pooling loss terms. Combining the two terms yields slightly improved denoising, as reported in the first two rows of Table 2 . The first, third, and fourth columns of Table 3 show the ML superpixel quality is similar, but the boundary recall is slightly worse when combining the two. The superpixel pooling quality increases by 2.33 dB PSNR when combined compared to only denoising. The second and last columns of Figure 8 qualitatively compare the pooled images.We observe a disharmony between the superpixel probabilities useful for denoising and those trained on segmentation labels. Table 2 reports a 2.6 dB PSNR drop in denoising quality. The first two rows of Table 3 report training with segmentation labels yields the best ML superpixel estimates, matching related works [8] . However, the superpixel pooling quality is poor. This suggests boundary detection is distinct from denoising and pixel pooling. memory. However, SNA's additional complexity does not scale with the number of features (F). ) and O(N F + HWK 2 ), respectively. Importantly, SNA's additional complexity does not scale with the number of features (F). Since the number of features is a significant factor in the computational cost of an attention operator, it is sensible to believe SNA can be used efficiently within large-scale deep neural networks. To be concrete, if the number of features is 128, then there is less than a 4% percent increase in FLOPs from NA to SNA.exp(λat q ⊺ i kj) O(F • HW • K 2 ) O(HW • K 2 ) O(F • HW • K 2 ) O(HW • K 2 ) exp(λat q ⊺ i kj) S s=1 π (i,s) π (j,s) O(9 • HW • K 2 ) O(2 • HW • K 2 ) wi,j = exp(λat q ⊺ i kj ) S s π (i,s) π (j,s) j′ ∈N (i) exp(λat q ⊺ i k j′ ) S s π (i,s) π (j′ ,s) O(HW • K 2 ) O(HW • K 2 ) O(HW • K 2 ) O(HW • K 2 ) j∈N (i) wi,jvj O(F • HW • K 2 ) O(F • HW) O(F • HW • K 2 ) O(F • HW) Total FLOPs & Peak Memory O([2F + 10] • HW • K 2 ) O(F • HW + 3 • HW • K 2 ) O([2F + 1] • HW • K 2 ) O(F • HW + 2 • HW • K 2 )A.1 Soft Superpixel Attention is the Optimal DenoiserThe latent superpixel model assumes the following data-generating process for each image pixel, Sample superpixel probabilities π ∼ p(π)Sample the superpixel assignment given the probabilities EQUATIONWe will construct a denoiser that depends on the superpixel probabilities rather than the superpixel assignment. Our derivation is inspired by [37] . Only the form of p(s i |π(i) ) = Categorical(s i ; π (i) )is known, and p(π) will not impact our final answer. So we only need to estimate the unknown density, p(x|s). To do this, we approximate p(x|s) from a sample (x, s, π),EQUATIONwhere p(s i |m) = π (m,si) . By re-weighting the empirical density with superpixel probabilities, rather than superpixel assignments, our estimate does not depend on the ambiguous superpixel labels. Notice if the superpixel probabilities are independent of the sample index, then the usual weighting is recovered, p(m|s i ) = p(m) = 1 M . Next, the pixels are corrupted with Gaussian noise,EQUATIONLet the denoiser be a function of the noise intensity and superpixel probabilities, D( xi ; σ, π). The expected error of the denoiser is taken with respect to the density pσ ( xi |s i )p(s i |π (i) )p(π),EQUATIONNote p(s i = s|π (i) ) → p(s|π (i) ) for clarity. The expectation can be minimized by minimizing each integrand for fixed a fixed noisy pixel ( x) and superpixel probability (π). Since the integrand is a sum of convex functions, the problem is convex and the solution is the point where the gradient is equal to zero.EQUATIONIn general, the optimal denoiser is written as,EQUATIONBy construction, the probability of a particular sample's superpixel assignment is written p(s|m) = π (m,s) . When the samples are restricted to a neighborhood surrounding pixel i, one can write the denoiser as the following expression,EQUATIONThis is soft superpixel neighborhood attention when the samples are restricted to a neighborhood surrounding pixel i, the qkv-transforms are identity, and the attention scale is fixed, λ at = 1 2σ 2 . Presently, we highlight where our per-pixel superpixel probabilities are used in the proof. In Equation 19, the probability p(s|π) depends on the sampled superpixel probabilities π. When denoising image pixel index i, we have p(s|π) = π (i,s) . However, under the standard model with a shape superpixel probability prior, p(s|π) does not depend on the particular pixel index. This reduces Equation 20 to a superpixel re-weighting term that does not depend on superpixel information from pixel i, which would be strange. We did experimentally try out this alternative re-weighting scheme, and its denoising quality matches NA. Finally, we discuss a conceptual rationale for sampling superpixel probabilities for each image pixel. The introduction (Sec 1) explains that superpixel assignment is an ambiguous classification task. There are many qualitatively "good" superpixel assignments, and often superpixel boundaries can be modified without impacting the overall quality. This suggests superpixel probabilities are more important than superpixel assignments. However, working with p(x|π) is more challenging than working with p(x|s). So in this paper, we still work with p(x|s) but design the denoiser to depend on the probabilities π to circumvent using the ambiguous superpixel assignments.Section 4.3 states that SNA is the optimal denoiser, but in practice, samples are limited to a neighborhood surrounding a query pixel. To analyze the impact of this restriction, this subsection computes the expected error over the image pixels and noise for a fixed superpixel sample. For expedient analysis, the number of features is one (F = 1, x i → x i ). The image pixels conditioned on the superpixel information are modeled with a Gaussian distribution, x i |s i , µ, η ∼ N (µ s , η 2 s ). We will analyze the attention modules by studying a denoiser of the following form,EQUATIONThis denoiser corresponds to using a flat attention scale λ at = 0 and using only prior superpixel probabilities as attention weights w j . Analysis of attention weights, in general, is difficult, and this approach removes the challenge. The expected error is computed over the Gaussian-corrupted and noise-free data density conditioned by the superpixel information, p( x|x)p(x|s, µ, η). The expected error is split into the classic bias-variance decomposition. See Section A.4 for the proof.The expected error of the denoiser D over the joint density p( x, x|s, µ, η) is written,EQUATIONThe NA module's flat weights achieve the minimum variance of 1 W σ 2 where W = |N (•)| is the window size. However, NA's bias depends entirely on the superpixel's neighbors and can be large if the neighborhood happens to contain pixels from superpixels with different means. For example, the bias will be large for pixels along the sharp orange edge in Figure 3 since their neighborhoods include the dissimilar blue region. Section A.3 shows that even when the attention scale is non-zero, NA cannot robustly reject dissimilar regions.H-SNA has the opposite problem of NA. H-SNA's sharp weights achieve the minimum possible bias of zero. However, H-SNA's variance depends entirely on the size of the superpixel. In Figure 3 , the snake-like superpixel reduces the number of included pixels, increasing the variance term of the error. This observation that thin and small superpixels are undesirable matches existing assumptions from superpixel literature [26, 5] .SNA's modulation of the superpixel weights allows it to be better than both NA and H-SNA. In Figure 3 , SNA excludes the dissimilar blue region (unlike NA) and includes the similar orange regions (unlike H-SNA). Said another way, SNA excludes only dissimilar regions.Section A.2 showed SNA yields a smaller theoretical error than NA because SNA can exclude dissimilar regions via superpixel probabilities. The previous analysis fixed the attention scale to zero (λ at = 0), and one may wonder if the attention scale might be used to exclude dissimilar regions as well. This subsection shows the attention scale excludes dissimilar pixels rather than dissimilar regions, which is less robust under noise. In this analysis, the qk-transforms are fixed to identity, the feature dimension is fixed to 1, and the distances function is the negative squared difference:d(y i , y j ) = -(y i -y j ) 2 .We first show the attention scale excludes dissimilar pixels rather than regions. Weights of pixels within a similar region should be similar, w i,j ≈ w i,i , while weights of pixels within dissimilar regions should be near zero, w i,j ≈ 0. The decay of dissimilar regions will only occur for NA when the scale increases (λ at > 0), but this growth also decays the weights associated with similar regions. Noting e λatd(yi,yi) = 1, the ratio of two weights for NA is as follows:EQUATION)Two noisy pixel values must be approximately equal for their attention weights to be approximately equal. In other words, excluding dissimilar regions with the attention weights also excludes nonidentical noisy pixels. This strict condition limits NA's ability to exclude dissimilar regions.Compare this with SNA, which we argue excludes regions more robustly. Say all but a single region should be excluded, so λ sp can be large. The inclusion of the single (useful) region depends on the most likely estimated superpixel matching the latent probabilities, s * i = ŝi . Using Section 3.1's formulation, the argmax terms match when the following condition is met,EQUATIONIn other words, SNA properly rejects dissimilar regions when a noisy pixel is more similar to its true mean than another superpixel mean. Comparing two noisy differences is more robust than comparing two noisy pixels. So we claim SNA rejects dissimilar regions more robustly than NA.This subsection presents a more detailed version of Claim 3 from Section A.2 and its proof.The expected error of the denoiser D over the joint density p( x, x|s, µ, η) is written,EQUATION)The following proves the above claim. The first step is to expand the square,EQUATION)Each expectation is evaluated separately,EQUATION= j∈N (i)EQUATION= j∈N (i)w 2 i,j (σ 2 + η 2 sj ) + j,k∈N (i) w i,j w i,k µ sj µ s k (30) E[D i ( x)x i ] = j∈N (i) w j E[x j x i ] = w i,i η 2 si + j∈N (i)w i,j µ si µ sj (31) Then all terms are put together and rearranged,E[(D i ( x) -x i ) 2 ] = η 2 si -2w i,i η 2 si + j∈N (i) w 2 i,j (σ 2 + η 2 sj ) + µ 2 si + j,k∈N (i) w j w k µ sj µ s k -2 j∈N (i) w i,j µ si µ sj (32) = η 2 si -2w i η 2 si + j∈N (i) w 2 j (σ 2 + η 2 sj ) + j,k∈N (i) w j w k µ 2 si + j,k∈N (i) w j w k µ sj µ s k -2 j,k∈N (i) w j w k µ si µ sj (33) = η 2 si (1 -2w i ) + j∈N (i) w 2 j (σ 2 + η 2 sj ) +   j∈N (i) w j (µ si -µ sj )   2 (34) Using E[X 2 ] = Var[X] + E[X] 2, the bias-variance terms can be identified. Since the E[X] 2 term is simple to compute and exactly matches the right-hand side, we can label those terms as the bias and assign the remaining terms to the variance. Increasing the neighborhood window size includes more samples to decrease the variance but also adds bias since the noisy samples can be weighted improperly. This bias-variance trade-off is illustrated by the increasing optimal window size as the noise intensity increases. Since the bias of H-SNA is nearly zero, the increasing neighborhood size only increases the denoising quality within the selected grid. Figure 11 : Number of Superpixels. The number of superpixels has a limited effect on the denoising quality when compared with other hyperparameters. Generally, increasing the number of superpixels increases the denoising quality. When the noise is low (σ = 10) the number of superpixels seems irrelevant. As the noise increases, the benefit of more superpixels becomes more apparent. Generally, using explicit SLIC iterations is more effective than predicting superpixel probabilities directly.This subsection explores the impact of hyperparameters related to the proposed SNA. This includes the neighborhood window size (Fig 9 ), the size of the auxiliary network ( Fig 10) , and the number of superpixels ( Fig 11) . All networks are trained and tested according to Section 5.2. Both the size of the auxiliary network and the neighborhood window size have a significant impact on the denoiser quality (over 1 dB). The number of superpixels has a smaller overall impact (less than 0.5 dB). NeurIPS Paper ChecklistQuestion: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The code is provided in the supplemental material and will be made publicly available upon acceptance. The code is implemented in Python using Pytorch, Numpy, Pandas, and CUDA and run using two NVIDIA Titan RTX GPUs and one RTX 3090 Ti GPU [40] [41] [42] [43] . The code includes a readme, and requirements files to setup the environment.Testing datasets are Set5 [44] , BSD100 [36] , Urban100 [45] , and Manga109 [46] .Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The experiment set-up described in Section 5.1 explains the important parameters used to run the denoising and superpixel-loss experiments. We train each network for 800 epochs using a batch size of 2 on the BSD500 dataset [36] using a learning rate of 2 • 10 -4 with a decay factor of 1/2 at epochs 300 and 600. These hyperparameters were chosen using an informal grid search. The code is provided in the supplemental material and will be made publicly available upon acceptance.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Error bars are not usually reported for the benchmarks used in this paper, and we believe including them will add unnecessary clutter to the report. If requested by the reviewers, the 95% confidence interval of the average PSNR/SSIM can be added to the experiments. However, generating error bars to assess the impact of training requires rerunning all of the experiments several times, which significantly increases the computational cost of this paper.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?• The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.Code for this project is available at https://github.com/gauenk/spix_paperFor educational value, hyperparameters are modified to create more boundaries.38th Conference on Neural Information Processing Systems (NeurIPS 2024).This paper presents the soft superpixel neighborhood attention (SNA) module as an alternative to neighborhood attention (NA). SNA accounts for deformable boundaries of objects. The key modeling assumption is that superpixel probabilities vary per pixel, and we find SNA is the optimal denoiser under this model. For a fixed budget of network parameters, SNA achieves a significantly better denoising quality than NA within a single-layer neural network. While SNA does require more computation than NA, SNA's computation does not scale with features which suggests SNA can be used efficiently within large-scale deep neural networks. 
 
 Appendix:  


Decision: The paper has received three strong support, but one borderline reject rating that has concerns about the technical novelty, significance and impact of the topic. But the reviewer did not elaborate the concerns and interact with the authors. More discussions, explanations and clarifications have been provided to largely the satisfactions of the relevant reviewers.

Reviews: review: paper_summary: paper_summary: This paper proposes an attention module in which the dot product weights are modified with
superpixel probabilities, named Superpixel Neighborhood Attention (SNA). By doing so, the
optimization process is arguably made easier by letting attention avoid learning spurious
interactions, which prior work into the attention module has found to be somewhat common. 
One key difference to prior works in superpixels is modelling superpixels as probabilities instead
of naive binary interactions, which in theory allows a flexible interpolation between standard
attention and superpixel attention. The proposed module is shown to improve the baseline approach
without superpixels by around 1 to 3 dB PSNR, and qualitative results support this claim as well.
strength_weakness: strength_weakness: 1. From my point of view, the most considerable weakness is a lack of discussion on the
computational complexity of the proposed approach. Note that I am not referring to the reported
performance gap, but generally to the fact that it is unclear from reading the paper how complex the
SNA attention algorithm is, and whether or not it can scale beyond the scope of the experiments done
in this paper. While metrics such as time and space complexity, and similar metrics such as FLOPs
don't necessarily translate into actual efficiency, they do provide a rough idea of whether or not
two operations are at least comparable in terms of resource requirement in theory. Given that the
baseline, neighborhood attention, is seemingly so much more efficient in runtime and memory usage,
and the fact that it has been performance optimized to some extent over time, and that these are
both operations in complex deep neural networks implemented primarily by a deep learning framework
with a different standard of implementation, I think the reported numbers are simply not that
informative. The disclosure and clarity is certainly appreciated, but stating that the proposed
approach's performance levels are "likely overly pessimistic" is not informative, and could be
easily reworded by providing some evidence that there isn't such a significant difference in
performance levels in theory.

2. Lack of comparison to other relevant approaches. I think limiting the use of dot product
attention to just one local pattern, namely neighborhood attention, is worth a discussion. Is there
a reason why self attention itself is not considered here, given that the superpixel probabilities
effectively serve as an implicit attention mask. It may also be true that one can map superpixel
probabilities directly to attention weights and define attention differently. I think while the
paper is mostly complete as is, this is something that merits at least a paragraph.

3. The citation for neighborhood attention is incorrect; the reference ("Neighbor2neighbor") does
not mention neighborhood attention. The correct reference is:

>Hassani, A., Walton, S., Li, J., Li, S. and Shi, H., 2023. Neighborhood attention transformer. 
>In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6185-6194).
questions: questions: 1. Is there any metric through which SNA's cost or resource requirement can be compared against
standard neighborhood attention with more certainty? The seemingly large performance gap in terms of
actual runtime and memory footprint, while necessary, is unsurprising, and only really reveals that
as part of future efforts, SNA could use performance optimization. However, it is completely unclear
how far that optimization could go. My suggestion is at the very least providing some other proxy
for that, be it in the form of time and space complexity, or any other relevant analysis of the
steps required by the algorithm and how easy / difficult they will be to parallelize and performance
optimize.

2. Can you elaborate Eq (1)? Softmax is an operation over a set or vector, and I would assume it
would be over N values per pixel, assuming each pixel has N superpixels.

3. This is just out of curiosity, but as of recently the neighborhood attention package does support
non-square kernels. Does that change in assumptions with regard to neighborhood attention change
anything about the significance of SNA over NA, or is that just a minor detail?

4. How are the efficiency performance metrics (runtime and memory usage) evaluated? Were standard
practices followed? (i.e. locking power and frequency, benchmarking without external factors such as I/O,
communication, framework-level optimizations such as caching allocators and memory planning,
measuring metrics iteratively and reporting an average, and the like)

5. Do either of SNA / H-SNA use the original neighborhood attention implementation, or is the
implementation done in this paper completely specific to superpixels? I'm trying to understand
whether SNA or its future application could also use additional performance optimizations done to
neighborhood attention, or whether it would require its own independent development and in turn face
similar issues in terms of extensibility as any other such approach.
limitations: limitations: While the reported performance gap is a limitation of the proposed approach at the moment, I think
it is very insignificant when taking into account the rest of the contributions of the paper.

However, I cannot be sure of that without evidence that would suggest the performance gap is simply
due to a lack of performance optimization. It could very well be the case that SNA in its current
form cannot scale easily beyond the scope of this paper, which would be in my opinion a far greater
limitation.

This is therefore either a limitation (meaning it is simply not possible or extremely difficult to
analyze SNA's computational cost and compare against standard neighborhood attention) or a weakness
(it was just not included in the paper.)
paper_summary: paper_summary: This paper proposes a soft superpixel neighborhood attention (SNA). It proves that SNA is the optimum denoiser under Gaussian noise. Experiments show that SNA outperforms other local attention modules for the image denoising task.
strength_weakness: strength_weakness: - Experiments are limited. The network that has been tested is written in Eq. (10).
- The current implementation is slow. Some effort is needed to program an efficient version, but this is left for future work.
questions: questions: I am not sure that I understand the training process. It is claimed that the network is trained on BSD500, but do you train by adding noise to the image, and targeting the network to produce the clean, original image?
limitations: limitations: The hypothesis for the theory are clearly stated, and the limitation are acknowledged.
paper_summary: paper_summary: The paper proposes a new attention mechanism based on Soft Superpixel Attention. The main idea is to use superpixel segmentations directly in the attention module. The paper proposes rigorous proof that the proposed mechanism is the optimal denoiser. Results show an improved denoising performance.
strength_weakness: strength_weakness: * The performance improvement is relatively small.
* The paper would be much stronger if it could show how this attention module could be used in other tasks than only on denoising. For instance, deblurring, object detection, tracking, etc.
questions: questions: How difficult is it to incorporate the proposed attention mechanism in other downstream tasks?
limitations: limitations: Yes, this is discussed.
paper_summary: paper_summary: The paper proposes soft superpixel neighborhood attention, with the motivation that attention is more efficient when it is local, and superpixels are better for local attention as patches, and furthermore, soft superpixels are more robust to errors than hard superpixels.
strength_weakness: strength_weakness: The technical novelty in the paper is not major.

The application (denoising) is not very appealing in the sense that it is not such an important application in the computer vision. It would be best to evaluate the proposed approach on some applications that are more widely used, such as semantic segmentation.

There is a theoretical contribution of optimality of the proposed denoiser is proven, but it assumes the proposed latent superpixel model, and it is not clear how true of the reality this proposed model is.
questions: questions: see above
limitations: limitations: Limitations are addressed.

Comments: title: Response
Reminder of the deadline
Paper discussion and rating finalization
comment: Thank you for the rebuttal. I maintain my accept rating.
Thanks for your response.

I maintain my previous score of "accept".
Dear Reviewers, 
While the deadline, Tuesday, 13th August, is approaching, can you please check at your early convenience the rebuttal from the authors, make or require further clarifications, if necessary, and interact with the authors over any further issues/concerns you may still have, and finalise the rating of the paper soon. Even though you have no further issues/concerns, you may want to acknowledge the responses to your questions in the rebuttal from the authors. 
Thank you very much for your time, effort and contribution to the organization of NeurIPS 2024,
Thank you; my questions have been resolved and I don't have any more.
I'm changing my confidence score and still vote for this paper to be accepted.
Dear Reviewers,
Can you please have a look at the reports of the other reviewers and also the rebuttal from the authors and respond to their questions, if available, then discuss any issues of your concern, finalise and reach a consensus about the rating of the paper before the deadline of the next Tuesday, 13th August? 
Thank you very much for your time, effort and contribution to the organization of NeurIPS 20024, 
AC