Title: Graph Convolutions Enrich the Self-Attention in Transformers!
Venue: NeurIPS 2024
Accepted: yes
Keywords: Transformer

self-attention

oversmoothing

graph filter
Abstract: Transformers, renowned for their self-attention mechanism, have achieved state-of-the-art performance across various tasks in natural language processing, computer vision, time-series modeling, etc. However, one of the challenges with deep Transformer models is the oversmoothing problem, where representations across layers converge to indistinguishable values, leading to significant performance degradation. We interpret the original self-attention as a simple graph filter and redesign it from a graph signal processing (GSP) perspective. We propose a graph-filter-based self-attention (GFSA) to learn a general yet effective one, whose complexity, however, is slightly larger than that of the original self-attention mechanism. We demonstrate that GFSA improves the performance of Transformers in various fields, including computer vision, natural language processing, graph-level tasks, speech recognition, and code classification.

Graph Convolutions Enrich the Self-Attention in Transformers!

Abstract

Transformers, renowned for their self-attention mechanism, have achieved state-ofthe-art performance across various tasks in natural language processing, computer vision, time-series modeling, etc. However, one of the challenges with deep Transformer models is the oversmoothing problem, where representations across layers converge to indistinguishable values, leading to significant performance degradation. We interpret the original self-attention as a simple graph filter and redesign it from a graph signal processing (GSP) perspective. We propose a graph-filter-based self-attention (GFSA) 1 to learn a general yet effective one, whose complexity, however, is slightly larger than that of the original self-attention mechanism. We demonstrate that GFSA improves the performance of Transformers in various fields, including computer vision, natural language processing, graph-level tasks, speech recognition, and code classification.

1 Introduction

Transformers are arguably one of the best feats in the field of deep learning. They are now showing state-of-the-art performance in various fields, ranging from computer vision to natural language processing, prediction tasks on graphs, speech recognition, and so forth [77, 16, 60, 61, 19, 74, 101, 45, 27, 41, 63, 51, 59, 38, 95, 72]. Recently, there have been several studies conducted on better understanding them [25, 3, 80]; there exists a common agreement among researchers that the self-attention is one of the keys leading to the success.

Understanding

(CoLA)

Modeling

(PTB)

Recognition

(LibriSpeech 960h)

Image

Classification

(ImageNet-1k)

Code

Classification

(Devign)

Graph

Regression

(ZINC)

0% 2%

4% 6%

6.25%

RoBERTa

0.31%

GPT2

4.55%

Transformer

1.63%

DeiT

2.40%

RoBERTa

4.03%

Graphormer

Backbones

+ GFSA

KAIST

-100

-50

0

50

100

Frequency

0.0

0.2

0.4

0.6

0.8

1.0

Normalized Magnitude

DeiT

DeiT + GFSA

(a) Filter response

1

6

12

18

24

Layer Index

0.2

0.4

0.6

0.8

Cosine Similarity

DeiT

DeiT + GFSA

(b) Cosine similarity

0

50

100

150

Singular Value Index

0.0

0.3

0.6

0.9

Normalized Singular Value

DeiT

DeiT + GFSA

(c) Singular value

However, there also exist several studies pointing out potential limitations of the self-attention [101, 18, 25, 28]. For instance, Shi et al. [71] revealed an analogy between the self-attention and the residual graph convolutional network (GCN), showing that BERT also suffers from a notorious problem of GCNs, called oversmoothing , i.e., tokens' latent representations become similar to each other at the deeper layers. In every self-attention layer, value vectors are aggregated in a weighted average manner since each row-wise sum of the attention matrix is always 1. Although each self-attention layer has its own attention matrix, this aggregation method causes the oversmoothing problem, not only in Transformers but also in graph neural networks [54, 7, 79, 66, 37, 101, 25, 92, 53, 71, 80, 3, 89, 90]. However, we confine our discussion to the oversmoothing of Transformers (see Section 2).

Being inspired by them, we redesign the self-attention from the perspective of graph signal processing (GSP) - in particular, we resort to GSP on directed graphs since the attention matrix is asymmetric. However, performing graph convolutions in the self-attention layer may incur non-trivial computational overheads. Therefore, our key design point is to learn a general but effective graph filter with minimal overhead. In general, a graph filter on a graph G is represented by a polynomial expression based on its adjacency or Laplacian matrix - in this regard, the existing self-attention mechanism can be understood as the simplest graph filter with ¯ A only, where ¯ A ∈ [0 , 1] n × n means a learned attention matrix and n is the number of input tokens.

Our proposed graph filter consists of an identity term and two matrix polynomial terms, ¯ A and ¯ A K . One can design better filters with more polynomial terms, but we avoid it since Transformers already require very large computation. The K -th power, ¯ A K , may also require a high computation when the number of tokens is large. To avoid this, we further approximate ¯ A K using the element-wise first-order Taylor approximation. Therefore, one can consider that our proposed graph filter is the very next complicated filter after the one used by the original self-attention mechanism. However, its efficacy is tremendous in various fields (cf. Fig. 1).

Our proposed filter enriches the self-attention with more diverse frequency information (see Fig. 2(a)) -low (resp. high) frequency signals on G mean neighboring nodes have similar (resp. different) values. Therefore, our method is able to not only effectively address the oversmoothing problem but also learn better latent representations for downstream tasks.

There exist a couple of prior works to enrich the self-attention mechanism with high frequency information [80, 4]. In comparison with them, our proposed graph filter is distinctive in the following aspects: i) our proposed filter is more effective and shows better performance with comparable computational overheads, ii) our proposed filter is well-aligned with recent advancements in the GCN community - in other words, some graph filters used by recent advanced GCN methods are special cases of our proposed graph filter, which is not the case for prior works, and iii) other methods were typically studied for certain domains only whereas we test our method in 6 domains - for instance, DiversePatch [25] works only for Vision Transformers (ViTs).

We replace the self-attention layer of selected Transformers in various fields with our proposed graph filter-based layer without changing other parts. Therefore, the accuracy increases in them are solely by our proposed graph filter-based self-attention. These enriched Transformers increase the model performance by 1.63% for image classification, 6.25% for natural language understanding, 0.31% for

causal language modeling, 4.03% for graph regression, 4.76% for speech recognition, and 2.40% for code classification (see Fig. 1). Our core contributions are as follows:

We provide a novel perspective on self-attention as a graph filter. This perspective allows us to design more effective self-attention that can address the oversmoothing problem.

We propose a graph filter-based self-attention (GFSA) mechanism, integrating an identity term and two polynomial terms for general yet effective than the simple self-attention mechanism (Section 3).

We demonstrate that GFSA improves the performance of Transformers on a variety of tasks. GFSA achieves improved results on natural language processing, computer vision, speech recognition, graph-level tasks, and code classification (Sections 5.1 to 5.6).

We devise a strategy to selectively apply GFSA to even-numbered layers, effectively mitigating the computational overhead while preserving GFSA's performance (Section 6).

2 Background & Related Work

2.1 Self-Attention in Transformers

The core building block of the Transformer architecture is the self-attention mechanism, which enables the model to learn attention patterns over its input tokens [77]. The self-attention mechanism, denoted as SA : R n × d → R n × d , can be expressed as follows:

where X ∈ R n × d is the input feature matrix, W key ∈ R d × d , W qry ∈ R d × d , and W val ∈ R d × d are the key, query, and value trainable parameters, respectively, and d is the dimension of each token. The self-attention mechanism allows the model to weigh the importance of each token in the input sequence relative to the others, enabling the model to capture long-range contextual information better. The Transformer architecture includes multiple layers, each with a multi-head self-attention layer followed by a position-wise feed-forward layer.

2.2 Self-Attention and Graph Convolutional Filter

The self-attention matrix used in Transformers has the form of symmetrically normalized adjacency matrix where each token become a node [71, 28] - the symmetrically normalized adjacency matrix is a special case of asymmetric (or directed) adjacency matrix where each row is normalized and is frequently used for the graph signal processing (GSP) on directed graphs [49]. A weighted graph G with adjacency matrix A can be constructed by using the input tokens as n nodes and the edge weights between node i and node j as exp(( XW qry ) ⊺ i ( XW key ) j ) . We can rewrite the self-attention matrix ¯ A ij as exp (( XW qry ) ⊺ i ( XW key ) j ) ∑ d k =1 exp ( XW qry ) ⊺ i ( XW key ) k . This allows ¯ A to be interpreted as the symmetrically normalized adjacency matrix. In other words, ¯ A = D -1 A , where D = diag ( d 1 , d 2 , . . . , d n ) and d i = ∑ j A i,j .

Our new attention method is designed on top of GSP which has a close connection to discrete signal processing (DSP) [67, 68]. In DSP, a discrete signal with a length of n can be represented by a vector x ∈ R n . Let g ∈ R n be a filter that we want to apply to x . The convolution x ∗ g can be written as follows:

where the index, denoted as i , refers to the i -th element in each vector.

GSP can be understood as a generalized concept of DSP. Signals are defined on the nodes of a graph, and the graph's structure influences signal processing operations. In addition, the linear and shift-invariant graph convolution filter H with n nodes can be written with a shift operator S as

follows S can be from a directed graph [48]:

where x ∈ R n is a 1-dimensional graph signal, K is the maximum order of polynomial, and w k ∈ [ -∞ , ∞ ] is a coefficient. S is an n × n matrix where ( i, j ) -th element is non-zero if and only if there is an edge from node i to j . Two representative samples of S are adjacency and Laplacian matrices. The graph filter H is the same as ∑ K k =0 w k S k with a large enough value of K , which is called matrix polynomial [48]. We note that this graph filtering operation can be extended to d -dimensional cases as in Eq. (1). Being inspired by Zou et al. [106] and Maskey et al. [49], we rely on the singular value domain analysis to understand the low/high-pass characteristics of filters on directed graphs (cf. Fig. 2). See more discussion in Appendices E and F.

In the context of the self-attention within Transformers, the core part of the self-attention in Eq. (1), i.e., ¯ AX , can be considered as a d -dimensional graph filter with ¯ A only, where H = ¯ A . Our goal in this paper is to design a simple (for computational purposes) yet effective form of H .

2.3 Oversmoothing in GCNs and Transformers

Oversmoothing is a phenomenon observed in deep learning models, especially in GCNs [39, 78]. As information is aggregated over multiple layers for multiple nodes (tokens), latent representations tend to become similar to each other, leading to a loss of distinctiveness in the representations [54, 104, 66].

Surprisingly, an oversmoothing-like phenomenon is also observed in Transformers [80, 71]. Unlike CNNs, Transformers can not benefit from simply deepening layers after a certain depth. Earlier studies hypothesize that this may be due to issues such as the attention or feature collapse or due to uniformity among patches or tokens [101, 25, 92]. Dong et al. [18] also point out that the output of a pure Transformer, i.e., an attention mechanism without skip connections or MLPs, tends to converge to a rank-1 matrix [18]. This analysis is followed by [53], which suggests that rank collapses incur vanishing gradients of attention queries and keys.

In this context, the self-attention acts as a low-pass filter, since the self-attention calculates the weighted average of the value vectors of tokens. Wang et al. [80, Theorem 1] also reveal that the self-attention is a low-pass filter, continuously reducing high-frequency information. This nature contributes to the oversmoothing phenomenon as unique high-frequency features are lost in deeper layers of the network, further worsening the uniformity of token representations. Therefore, we extend the term 'oversmoothing' to describe the degeneration challenge observed in Transformers.

There have been proposed many empirical countermeasures for ViT, such as patch diversification [102, 25], rank collapse alleviation [101, 99], and training stabilization [74, 98]. Similar alleviating methods have been also proposed in the field of NLP, such as unsupervised learning [9], and resolve the oversmoothing and the token uniformity (or information diffusion) problems [18, 92]. There are studies on utilizing high frequency information via frequency domain analyses [80, 4], but they are not designed on top of graph filtering perspectives. Dovonon et al. [20] find that Transformers are not inherently low-pass filters, but oversmoothing depends on the eigenspectrum of the self-attention layers. They propose a reparametrization of the Transformer weights, ensuring that oversmoothinng does not occur.

Our paper addresses the oversmoothing problem with graph filters since the self-attention mechanism is a basic graph filtering operation as seen in the previous subsection.

3 Graph Filter-based Self-Attention Layers

Let ¯ A ∈ [0 , 1] n × n , where n is the number of tokens in the input to the self-attention layer, be a self-attention matrix. Since Transformers use multi-head self-attentions, there are multiple such matrices. For simplicity, but without loss of generality, we discuss only one head in one layer.

From the GSP perspective, using ¯ A as the shift operator, a graph filter can be represented as a matrix polynomial filter, as mentioned in Section 2.2. We aim to design this matrix polynomial filter using the two lowest-order terms and one high-order term in Eq. (3). The following theorem shows that,

despite using the three terms, the filter can be either a low-pass filter or a high-pass filter, depending on the coefficient values.

Theorem 3.1 (Filter characteristics based on coefficient values) . Let ¯ A be a self-attention matrix interpreted as a graph with connected components. Consider the polynomial graph filter defined by ∑ K k =0 w k ¯ A k , where w 2 , w 3 , . . . , w K -1 = 0 and only w 0 , w 1 , and w K are non-zero. If the coefficients w k for k = 0 , 1 , K are positive and their sum is 1, then the polynomial filter acts as a low-pass filter, attenuating high-frequency components and promoting smoothness across the graph. Conversely, if w k = ( -α ) k for k = 0 , 1 , K and α ∈ (0 , 1) with sufficient large K , the polynomial filter exhibits high-pass filter behavior.

The proof of Theorem 3.1 is in Appendix G. Based on Theorem 3.1, we propose to use the following graph filter, H GFSA, where the two lowest-order terms and one high-order term of the matrix polynomial are used:

where w 0 , w 1 , w K are coefficients and K is a hyper-parameter where K ≥ 2 . The coefficients can be learnable weights and we learn them with gradient descent algorithms.

Approximation of the high-order term. In Eq. (4), it is costly to calculate ¯ A K when K is large, so we need a way to approximate the high-order term ¯ A K in GFSA. We use the first-order Taylor approximation at point a = 1 for this purpose:

thus, we approximate f ( K ) = ¯ A K as follows:

Computing the derivative of ¯ A K directly at the evaluation point requires high computational costs. To overcome this problem, we adopt the forward finite difference method, which approximates derivatives with the difference term:

where the approximation error is O ( h 2 ) . To balance the trade-off between computational efficiency 2 and accuracy, we set h = 1 . This method is inspired by the approach in Brouwer et al. [6], which uses the difference term between two consecutive hidden states in discrete time to approximate the derivatives. Therefore, we approximate ¯ A K as:

The approximation for ¯ A K with ¯ A and ¯ A 2 provides a simpler computation that can significantly reduce the required computational resources and time.

GFSA: our graph filter-based self-attention. Our proposed graph filter-based self-attention (GFSA) is defined with the graph filter ˜ H GFSA as follows:

where the last term is the approximated ¯ A K from Eq. (8). We replace the original self-attention layer in various Transformers with the proposed graph filter-based layer without changing other parts. Therefore, GFSA can be plugged into any Transformers that rely on the self-attention. For pseudocode, see Appendix I.

4 Properties of GFSA

This section analyzes the theoretical error of the ¯ A K approximation used by GFSA and how GFSA can mitigate oversmoothing. We also explain the meaning of GFSA's high-order term in the context of Transformers and provide comparisons of GFSA in other models.

Theoretical characteristics of approximation error in GFSA. Weprovide a theorem that provides an upper bound on the error introduced by approximating the power of a matrix, specifically using the first-order Taylor expansion. The following theorem specifically analyzes the error of matrix ¯ A K when approximated using a first-order Taylor expansion.

Theorem 4.1 (Error bound for approximated high-order term in GFSA) . Define the error term, E K , as the difference between the exact value and approximated value of ¯ A K , which is given by E K = || ¯ A K -( ¯ A +( K -1)( ¯ A 2 -¯ A )) || F , where || · || F denotes the Frobenius norm. Then, the error bound can be shown that E K ≤ 2 √ nK.

The error bound provides an upper limit for the difference between the actual value of ¯ A K and its approximation. The proof of Theorem 4.1 is in Appendix H. It shows theoretical validity for using approximations to the high-order term in the filters of our GFSA. In terms of performance, we report a difference of approximately ¯ A K between the actual calculated values in Appendix J.

How to alleviate the oversmoothing problem? The key leading to the low/high pass filtering behavior of our proposed filter is the coefficients { w 0 , w 1 , w K } -note that in the self-attention of Transformers, w 0 = w K = 0 and w 1 = 1 . Since our method can learn any appropriate values for them for a downstream task, it can be reduced to low-pass-only, high-pass-only, or combined filters. According to Theorem 3.1, our graph polynomial filter can be said to be a low-pass filter when w 1 , w K are positive and a high-pass filter when they are negative. Therefore, our method can learn the appropriate coefficients { w 0 , w 1 , w K } for downstream tasks, so it can be reduced to a low-pass-only, high-pass-only, or combined filter, alleviating the oversmoothing problem of self-attention.

The meaning of the high-order term in GFSA in the context of Transformers. Existing selfattention only captures simple pairwise similarities between tokens and is limited in capturing high-order dependencies. For example, given the two sentences, 'Books are more expensive than pencils' and 'Books are cheaper than computers', to understand the relationship between 'computers' and 'pencils', we need to capture the high-order dependencies connected through the 'Book' token. However, it is difficult to capture these high-order dependencies with traditional self-attention [103]. Therefore, from a Transformer perspective, the approximated ¯ A K in GFSA can be interpreted as being able to capture these high-order dependencies.

Comparison to Transformers. In the field of computer vision, there has been recent research on adjusting the frequency response of ViT. HAT [4] creates adversarial examples by altering clean images with high-frequency perturbations and jointly trains the ViT on clean images and adversarial examples. Through this, they aim to solve the problem of the ViT being unable to capture highfrequency by allowing us to capture the high-frequency components of the images. However, HAT has the disadvantage of requiring more epochs than the existing ViT, as it must perform adversarial training in some initial epochs and train normally in the remaining epochs. Wang et al. [80] use the concept of DSP, which is a special case of GSP, to isolate the lowest frequency component in the Fourier domain and use a filter learned by rescaling the low and high-frequency components. On the other hand, our GFSA extends the concept to graph signal processing and redesigns self-attention as a graph filter. While GFSA seeks to design a better graph filter by interpreting self-attention as a graph filter, Shi et al. [71] are inspired by JKNet [91], and they solve the oversmoothing problem by fusing the hidden vectors of each layer. However, their method has a limitation with memory increasing, and they only applied it to BERT.

Comparison to GCNs. Comparisons to GCNs that can be interpreted as graph filters [39, 15, 24] are inevitable. GFSA without a high-order term is analogous to ChebNet [15] with K = 1 . In addition, GFSA reduces to the vanilla GCN [39] when K = 1 , w 0 = 0 , w 1 = 1 . GPR-GNN [12], which approximates graph convolutions using the monomial basis, is identical to GFSA if it only considers up to first order and additionally uses a K -order term and learns the coefficients. When we

use only a high-order term and w K is learned to a negative value, GFSA can become similar to the reaction-diffusion layer of GREAD [13], ¯ AX + β ( ¯ A -¯ A 2 ) , depending on the higher order terms.

5 Experiments

In this section, we demonstrate the effectiveness of GFSA through a series of experiments. These experiments encompass various tasks: i) language understanding and causal language modeling, ii) image classification, iii) graph-level task, and iv) code classification. We replace the self-attention of base Transformers in those fields with our GFSA. Our modification adds only tens to hundreds of parameters, which are negligible in comparison with the original size of base models.

5.1 Experiments on Natural Language Understanding

Setting. We integrate GFSA into 3 pre-trained large language models: BERT, ALBERT, and RoBERTa. We evaluate them on the GLUE benchmark, which includes 3 categories of natural language understanding tasks: i) single-sentence, ii) similarity and paraphrasing, and iii) natural language inference tasks. For each task, we select the best hyperparameters for GFSA, and the other hyperparameters are fixed. The detailed experimental settings are in Appendix K.1.

Results. The results are shown in Table 1. When GFSA was plugged into backbones, average performance scores improved across all models over pure backbones. This indicates that GFSA is effective in both large models like BERT and RoBERTa, as well as relatively smaller models like ALBERT. It is worth mentioning that in the case of RoBERTa finetuned on the CoLA dataset; there is a significant margin increase from 60.34% to 64.11%, which is a 3.77% improvement with only 144 additional parameters. When compared to ContraNorm, GFSA shows a greater performance improvement on average. Fig. 5 in Appendix C shows that these performance enhancements can be attributed to addressing the oversmoothing issue through the designed graph filter.

5.2 Experiments on Causal Language Modeling

Setting. We also validate the effectiveness of GFSA on causal language modeling problems. We finetune GPT2 [61] on the following 3 datasets: Penn Treebank (PTB) [47], WikiText-2, and WikiText103 [50]. Following the evaluation method in Yao et al. [93], we finetune models for 15 epochs with PTB, 4 epochs with WikiText-103, and 10 epochs with WikiText-2, and report the perplexity for sensitivity metric. The detailed experimental settings are in Appendix L.1.

Results. Table 2 shows the perplexity on PTB, WitiText-2, and WikiText-103. Across all datasets, GPT2 with GFSA consistently outperforms the vanilla GPT2. Our GFSA improves the average perplexity from 18.806 to 18.764. Note that performance improvements

are made with only 144 additional learnable parameters for 12 layers with 12 heads.

5.3 Experiments on Vision Transformers

Setting. We aim to demonstrate the efficacy of our GFSA across a spectrum of ViT backbones. We choose DeiT [74], CaiT [75], and Swin [45] as the backbone, and the models are trained from scratch. When training the 12-layer DeiT, we follow the same training recipe, hyperparameters, and data augmentation from Touvron et al. [74]. For detailed experimental settings, see Appendix M.1.

Results. The experimental evaluations are summarized in Table 3. We compare various models on the ImageNet-1k benchmark. The results show that the proposed GFSA successfully enhances DeiT, CaiT, and Swin across all depth settings and training methods. GFSA provides additional parameters less than 72 for 12-layer DeiT while improving top-1 accuracy by 1.63%. To sum up, we observed that both shallow and deep ViTs can achieve the following benefits from GFSA: i) The filter response shows GFSA can preserve higher-frequency representation (cf. Fig. 2 (a)) and ii) Fig. 2 (b) shows that GFSA mitigates the increase in the cosine similarity of representation as the layer gets deeper. We further compare with state-of-the-art models that use Fourier transforms rather than graph filters in Appendix M.5. We also show results under the same settings as ContraNorm [28] in Appendix M.6.

5.4 Experiments on Graph-level Tasks

Setting. To evaluate the efficacy of GFSA on graph-level tasks, we conduct experiments on a broader range of datasets. We use datasets from Long-Range Graph Benchmark (LRGB) [21] (e.g., Peptide-func and Peptide-struct), Benchmarking GNNs [22] (e.g., ZINC, MNIST, CIFAR10), Open Graph Benchmark (OGB) dataset [32] (e.g., Molhiv and MolTox21), and OGB-LSC dataset (i.e., PCQM4M-LSC) [33]. We choose Graphormer [94], Graph-ViT [30], and GPS [63] as our backbone architectures, following their original experimental protocols for fair comparison. For GPS, we replace its self-attention module with our GFSA while maintaining its best configuration and other hyperparameters. For Graph-ViT, we apply GFSA to the Hadamard self-attention method, which He et al. [30] propose as optimal. For a detailed experimental setting, see Appendix O.1.

Results. Tables 4, 5, and 6 show consistent performance improvements when GFSA is integrated with backbone architectures. Graph-ViT + GFSA shows improvements on all datasets. On Peptidefunc, it achieves a 0.65% increase in AP. Notably, in PCQM4M, incorporating GFSA improves the validation MAE by 7.20%. Due to space constraints, the results with standard deviation are included in Appendix O.2.

5.5 Experiments on Automatic Speech Recognition

Setting. We conduct automatic speech recognition (ASR) experiments on the LibriSpeech 3 dataset [55], which consists of audio recordings paired with their transcriptions. We use Branch-

former [59] and a pure Transformer. For implementation, we follow the recipes of SpeechBrain [65] and the detailed settings are in Appendix N.1.

Results. Table 7 compares word error rates (WERs) on LibriSpeech 100h and 960h. For 100h, Transformer+GFSA achieves 10.30/25.30 on the test clean/other set, which is a 6.53% improvement over the Transformer for the WER of the test clean. For 960h, Transformer+GFSA shows a WER result of 2.31 in test clean, a 4.55% improvement over Transformer and Branchformer+GFSA achieves 2.31/5.49 with an LM on the test clean/other sets. Fig. 8 in Appendix N.2 depicts the learning curves of train loss and valid loss when using GFSA, showing the effectiveness of our proposed filter.

5.6 Experiments on Code Classification

Setting. We conduct a code defect detection task based on Devign dataset provided by Zhou et al. [105]. We use RoBERTa [44], CodeBERT [23], PLBART [2], and CodeT5 [84] as our backbone models. The detailed settings are in Appendix P.1.

Results. Table 8 shows the accuracy of all models; GFSA results better than the base models. The biggest improvement is 2.40% for RoBERTa. In the case of CodeT5-base, using GFSA shows an accuracy of 64.75, an improvement of 1.95% from 63.51. CodeT5small+GFSA has only about 100 additional parameters compared to CodeT5-small with 60M parameters, and even more impressively, it surpasses the accuracy of CodeT5-base. The biggest improvement is 2.40% for RoBERTa. In Appendix P.2, we include case studies for this task. We also report the results of the code clone detection task in Appendix Q.

6 Discussion on Runtime Overheads

Limitation. The introduction of our GFSA layer results in a slight increase in training and inference time. We report the runtimes when plugging GFSA in Appendices R and S. For GLUE benchmark, integrating GFSA into BERT enhances the average performance from 82.51% to 83.58% (see Table 1) with more overhead of less than 36 seconds per epoch based on average training time (see Table 23). Considering the improvements, the increases in training time are negligible.

GFSA in selected layers: a strategy to mitigate the limi-

tation. As GFSA requires more calculation than the original self-attention, the runtime after using GFSA slightly increases. Our experiments initially applied GFSA across all Transformer layers (as discussed in Section 5); however, to reduce computational load, we propose a selective application strategy. For this purpose, GFSA is used only on even-numbered layers. In Tables 35 to 39 of Appendix T, the results show that this strategy effectively reduces runtime increases while preserving comparable performance to the full-layer GFSA integration. Notably, the selective application of GFSA cuts the per-epoch runtime increase by 26.90% relative to its full-layer application, with only a 7.39% increase in runtime per epoch compared to the backbone model in Table 39.

GFSA in linear Transformers. Although GFSA requires additional computation for calculating ¯ A 2 , we explore integrating GFSA with linear attention variants to maintatin efficiency and scalability. Recent approaches [36, 70] achieve linear complexity by reformulating softmax operations and reordering matrix multiplication in self-attention. We apply similar principles to compute second-order self-attention efficiently, enabling ˜ H GFSA calculation with linear complexity with respect to sequence length. Fig. 4 shows the performance, runtime and GPU usage changes when applying our GFSA to Transformers with linear complexity. GFSA still improves performance compared to the backbone model, while the increase in time and GPU usage is minimal. Notably, when GFSA is applied to Efficient Attention [70], the performance is improved and the runtime is 11.82 times faster

DeiT-S

+ GFSA

+ GFSA

79.5

80.0

80.5

81.0

81.5

Top-1 Accuracy (%)

79.8

81.1

81.0

500

600

700

800

Runtime (s per epoch)

551

814

595

even

36.8

37

37.2

37.4

37.6

37.8

38

Accuracy (%)

0

100

200

300

400

500

600

700

800

Runtime (

s

per 1000 steps)

Trans.

Trans.

+ GFSA

Lin. YOSO-E

Eff. Attn.

Eff. Attn.

+ GFSA

11.82x

Faster

than when GFSA is applied to the vanilla self-attention. This shows that GFSA can be effectively implemented with linear complexity architectures while preserving its benefits and providing a solution for addressing computational concerns.

7 Conclusion

Our proposed GFSA achieves high performance with improvements on a variety of tasks. GFSA is a simple yet effective method that enriches self-attention in Transformers with more diverse frequency information. This enables GFSA to address the oversmoothing problem and learn better latent representations for downstream tasks. However, our GFSA does not bring significant overheads in those Transformers' empirical runtime complexity. One can use more complicated graph filters to enhance accuracy more, but our goal is to find a balance between accuracy enhancements and overheads in runtime complexity.

We believe that GFSA suggests a promising new direction for improving Transformers. GFSA can be implement with simple way and used in conjunction with other techniques to further improve the performance of Transformers. Considering the ongoing advancements in large language models, such as GPT-4 [1] and LLaMA [76], we hope that our approach may offer new insights for enhancing their performance and efficiency.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Unified pre-training for program understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2655-2668, 2021.

Ameen Ali, Tomer Galanti, and Lior Wolf. Centered self-attention layers. arXiv preprint arXiv: 2306.01610 , 2023.

Jiawang Bai, Li Yuan, Shu-Tao Xia, Shuicheng Yan, Zhifeng Li, and Wei Liu. Improving vision transformers by revisiting high-frequency components. In European Conference on Computer Vision , pages 1-18. Springer, 2022.

Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC , 7:8, 2009.

Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Chen Cai and Yusu Wang. A note on over-smoothing for graph neural networks. arXiv preprint arXiv:2006.13318 , 2020.

Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055 , 2017.

Nuo Chen, Linjun Shou, Ming Gong, Jian Pei, Bowen Cao, Jianhui Chang, Daxin Jiang, and Jia Li. Alleviating over-smoothing for unsupervised sentence representation. arXiv preprint arXiv:2305.06154 , 2023.

Tianlong Chen, Zhenyu Zhang, Yu Cheng, Ahmed Awadallah, and Zhangyang Wang. The principle of diversity: Training stronger vision transformers calls for reducing all levels of redundancy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12020-12030, 2022.

Zihan Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. Quora question pairs, 2018.

Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized PageRank graph neural network. In Proceedings of the International Conference on Learning Representations (ICLR) , 2021.

Jeongwhan Choi, Seoyoung Hong, Noseong Park, and Sung-Bae Cho. Gread: Graph neural reaction-diffusion networks. In International Conference on Machine Learning (ICML) , pages 5722-5747. PMLR, 2023.

George Dasoulas, Kevin Scaman, and Aladin Virmaux. Lipschitz normalization for selfattention layers with application to graph neural networks. In International Conference on Machine Learning (ICML) , pages 2456-2466. PMLR, 2021.

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems (NeurIPS) , 2016.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423 .

Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005) , 2005.

Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning (ICML) , pages 2793-2803. PMLR, 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations (ICLR) , 2021.

Gbètondji JS Dovonon, Michael M Bronstein, and Matt J Kusner. Setting the record straight on transformer oversmoothing. arXiv preprint arXiv:2401.04301 , 2024.

Vijay Prakash Dwivedi, Ladislav Rampášek, Mikhail Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. Long range graph benchmark. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022. URL https://openreview.net/forum?id=in7XC5RcjEn .

Vijay Prakash Dwivedi, Chaitanya K Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. Journal of Machine Learning Research , 24(43):1-48, 2023.

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 1536-1547, 2020.

Johannes Gasteiger, Stefan Weißenberger, and Stephan Günnemann. Diffusion improves graph learning. In Advances in Neural Information Processing Systems (NeurIPS) , volume 32, 2019.

Chengyue Gong, Dilin Wang, Meng Li, Vikas Chandra, and Qiang Liu. Vision transformers with patch diversification. arXiv preprint arXiv:2104.12753 , 2021.

Alex Graves and Navdeep Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In International Conference on Machine Learning (ICML) , pages 1764-1772. PMLR, 2014.

Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100 , 2020.

Xiaojun Guo, Yifei Wang, Tianqi Du, and Yisen Wang. ContraNorm: A contrastive learning perspective on oversmoothing and beyond. In Proceedings of the International Conference on Learning Representations (ICLR) , 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Xiaoxin He, Bryan Hooi, Thomas Laurent, Adam Perold, Yann LeCun, and Xavier Bresson. A generalization of vit/mlp-mixer to graphs. In International conference on machine learning (ICML) , pages 12724-12745. PMLR, 2023.

Nicholas J Higham. Functions of matrices: theory and computation . SIAM, 2008.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems (NeurIPS) , 33:22118-22133, 2020.

Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. OGBLSC: A large-scale challenge for machine learning on graphs. arXiv preprint arXiv:2103.09430 , 2021.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4700-4708, 2017.

John J Irwin, Teague Sterling, Michael M Mysinger, Erin S Bolstad, and Ryan G Coleman. Zinc: a free tool to discover chemistry for biology. Journal of chemical information and modeling , 52(7):1757-1768, 2012.

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning , pages 5156-5165. PMLR, 2020.

Nicolas Keriven. Not too little, not too much: a theoretical analysis of graph (over) smoothing. In Advances in Neural Information Processing Systems (NeurIPS) , volume 35, pages 22682281, 2022.

Jayoung Kim, Yehjin Shin, Jeongwhan Choi, Hyowon Wi, and Noseong Park. Polynomialbased self-attention for table representation learning. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 24509-24526. PMLR, 21-27 Jul 2024. URL https://proceedings.mlr.press/v235/kim24ae.html .

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In Proceedings of the International Conference on Learning Representations (ICLR) , 2017.

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019.

Siddique Latif, Aun Zaidi, Heriberto Cuayahuitl, Fahad Shamshad, Moazzam Shoukat, and Junaid Qadir. Transformers in speech processing: A survey. arXiv preprint arXiv:2303.11607 , 2023.

James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprint arXiv:2105.03824 , 2021.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 7871-7880, 2020.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. 1993.

Antonio G. Marques, Santiago Segarra, and Gonzalo Mateos. Signal processing on directed graphs: The role of edge directionality when processing and learning from network data. IEEE Signal Processing Magazine , 37(6):99-116, 2020. doi: 10.1109/MSP.2020.3014597.

Sohir Maskey, Raffaele Paolino, Aras Bacho, and Gitta Kutyniok. A fractional graph laplacian approach to oversmoothing. In Advances in Neural Information Processing Systems (NeurIPS) , 2023.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843 , 2016.

Luis Müller, Mikhail Galkin, Christopher Morris, and Ladislav Rampášek. Attending to graph transformers. Transactions on Machine Learning Research , 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=HhbqHBBrfZ .

Tam Nguyen, Tan Nguyen, and Richard Baraniuk. Mitigating over-smoothing in transformers via regularized nonlocal functionals. In Advances in Neural Information Processing Systems (NeurIPS) , volume 36, 2023.

Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. In Advances in Neural Information Processing Systems (NeurIPS) , volume 35, pages 27198-27211, 2022.

Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. In Proceedings of the International Conference on Learning Representations (ICLR) , 2020.

Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages 5206-5210. IEEE, 2015.

Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. Interspeech 2019 , 2019.

Badri Patro and Vijay Agneeswaran. Scattering vision transformer: Spectral mixing matters. In Advances in Neural Information Processing Systems (NeurIPS) , volume 36, 2023.

Badri N Patro, Vinay P Namboodiri, and Vijay Srinivas Agneeswaran. Spectformer: Frequency and attention is what you need in a vision transformer. arXiv preprint arXiv:2304.06446 , 2023.

Yifan Peng, Siddharth Dalmia, Ian Lane, and Shinji Watanabe. Branchformer: Parallel mlp-attention architectures to capture local and global context for speech recognition and understanding. In International Conference on Machine Learning (ICML) , pages 17627-17643. PMLR, 2022.

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research , 21(1):5485-5551, 2020.

Ladislav Rampášek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. In Advances in Neural Information Processing Systems (NeurIPS) , volume 35, pages 14501-14515, 2022.

Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. In Advances in Neural Information Processing Systems (NeurIPS) , volume 34, pages 980-993, 2021.

Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju-Chieh Chou, Sung-Lin Yeh, Szu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidong Na, Yan Gao, Renato De Mori, and Yoshua Bengio. SpeechBrain: A general-purpose speech toolkit, 2021. arXiv:2106.04624.

T. Konstantin Rusch, Michael M. Bronstein, and Siddhartha Mishra. A survey on oversmoothing in graph neural networks. arXiv preprint arXiv: Arxiv-2303.10993 , 2023.

Aliaksei Sandryhaila and José MF Moura. Discrete signal processing on graphs. IEEE transactions on signal processing , 61(7):1644-1656, 2013.

Aliaksei Sandryhaila and Jose MF Moura. Discrete signal processing on graphs: Frequency analysis. IEEE Transactions on Signal Processing , 62(12):3042-3054, 2014.

Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green AI. Communications of the ACM , 63(12):54-63, 2020.

Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 3531-3539, 2021.

Han Shi, Jiahui Gao, Hang Xu, Xiaodan Liang, Zhenguo Li, Lingpeng Kong, Stephen Lee, and James T Kwok. Revisiting over-smoothing in bert from the perspective of graph. In Proceedings of the International Conference on Learning Representations (ICLR) , 2022.

Yehjin Shin, Jeongwhan Choi, Hyowon Wi, and Noseong Park. An attentive inductive bias for sequential recommendation beyond the self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 8984-8992, 2024.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing , pages 1631-1642, 2013.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning (ICML) , pages 10347-10357. PMLR, 2021.

Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 32-42, 2021.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems (NeurIPS) , volume 30, 2017.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. In Proceedings of the International Conference on Learning Representations (ICLR) , 2018.

Guangtao Wang, Rex Ying, Jing Huang, and Jure Leskovec. Multi-hop attention graph neural network. In IJCAI , 2021.

Peihao Wang, Wenqing Zheng, Tianlong Chen, and Zhangyang Wang. Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice. In Proceedings of the International Conference on Learning Representations (ICLR) , 2022.

Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 , 2020.

Wei Wang, Ming Yan, and Chen Wu. Multi-granularity hierarchical attention fusion networks for reading comprehension and question answering. arXiv preprint arXiv:1811.11934 , 2018.

Wenhan Wang, Ge Li, Bo Ma, Xin Xia, and Zhi Jin. Detecting code clones with graph neural network and flow-augmented abstract syntax tree. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER) , pages 261-271, 2020. doi: 10.1109/SANER48275.2020.9054857.

Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 86968708, 2021.

Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics , 7:625-641, 2019.

Ross Wightman. Pytorch image models. https://github.com/rwightman/ pytorch-image-models , 2019.

Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426 , 2017.

Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. CvT: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 22-31, 2021.

Xinyi Wu, Amir Ajorlou, Zihui Wu, and Ali Jadbabaie. Demystifying oversmoothing in attention-based graph neural networks. In Advances in Neural Information Processing Systems (NeurIPS) , 2023.

Xinyi Wu, Zhengdao Chen, William Wei Wang, and Ali Jadbabaie. A non-asymptotic analysis of oversmoothing in graph neural networks. In Proceedings of the International Conference on Learning Representations (ICLR) , 2023.

Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International Conference on Machine Learning (ICML) , pages 5453-5462, 2018.

Hanqi Yan, Lin Gui, Wenjie Li, and Yulan He. Addressing token uniformity in transformers via singular value transformation. In Uncertainty in Artificial Intelligence , pages 2181-2191. PMLR, 2022.

Zhewei Yao, Xiaoxia Wu, Conglong Li, Connor Holmes, Minjia Zhang, Cheng Li, and Yuxiong He. Random-ltd: Random and layerwise token dropping brings efficient training for large-scale transformers. arXiv preprint arXiv:2211.11586 , 2022.

Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? In Advances in Neural Information Processing Systems (NeurIPS) , volume 34, pages 28877-28888, 2021.

Youn-Yeol Yu, Jeongwhan Choi, Woojin Cho, Kookjin Lee, Nayong Kim, Kiseok Chang, ChangSeung Woo, ILHO KIM, SeokWoo Lee, Joon Young Yang, SOOYOUNG YOON, and Noseong Park. Learning flexible body collision dynamics with hierarchical contact mesh transformer. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=90yw2uM6J5 .

Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision , pages 558-567, 2021.

Zhanpeng Zeng, Yunyang Xiong, Sathya Ravi, Shailesh Acharya, Glenn M Fung, and Vikas Singh. You only sample (almost) once: Linear cost self-attention via bernoulli sampling. In International conference on machine learning , pages 12321-12332. PMLR, 2021.

Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M. Susskind. Stabilizing transformer training by preventing attention entropy collapse. In Proceedings of the 40th International Conference on Machine Learning (ICML) , volume 202, pages 40770-40803. PMLR, 2023.

Aston Zhang, Alvin Chan, Yi Tay, Jie Fu, Shuohang Wang, Shuai Zhang, Huajie Shao, Shuochao Yao, and Roy Ka-Wei Lee. On orthogonality constraints for transformers. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) , pages 375-382, 2021.

Ying Zhang, Mohammad Pezeshki, Philémon Brakel, Saizheng Zhang, César Laurent, Yoshua Bengio, and Aaron Courville. Towards end-to-end speech recognition with deep convolutional neural networks. Interspeech 2016 , 2016.

Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. DeepViT: Towards deeper vision transformer. arXiv preprint arXiv:2103.11886 , 2021.

Daquan Zhou, Yujun Shi, Bingyi Kang, Weihao Yu, Zihang Jiang, Yuan Li, Xiaojie Jin, Qibin Hou, and Jiashi Feng. Refiner: Refining self-attention for vision transformers. arXiv preprint arXiv:2106.03714 , 2021.

Haoyi Zhou, Siyang Xiao, Shanghang Zhang, Jieqi Peng, Shuai Zhang, and Jianxin Li. Jump self-attention: Capturing high-order statistics in transformers. In Advances in Neural Information Processing Systems (NeurIPS) , volume 35, pages 17899-17910, 2022.

Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. AI open , 1:57-81, 2020.

Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. In Advances in neural information processing systems (NeurIPS) , volume 32, 2019.

Chunya Zou, Andi Han, Lequan Lin, and Junbin Gao. A simple yet effective svd-gcn for directed graphs. arXiv preprint arXiv:2205.09335 , 2022.

Appendix

A Reproducibility Statement

To ensure the reproducibility and completeness of this paper, we include the Appendix with 12 sections. Appendix I provides our PyTorch-style pseudo code for our GFSA method. The pseudo code helps to implement our GFSA to any Transformers used a pure self-attention. All experiments in the paper are reproducible with additional implementation details provided in Appendices K to Q.

B Broader Impact

In terms of the broader impact of this research on society, we do not see the very negative impacts that might be expected. However, this paper may have implications for the carbon footprint and accessibility of learning algorithms. The computations required for machine learning research are rapidly growing, resulting in a larger carbon footprint [69]. Our study improves performance and increases runtime very slightly, but the runtime increase is not very significant. However, in future research, it will also be important to study and improve our GFSA by taking carbon footprints into account.

GFSA improves the performance of existing Transformer-based models, which can have many positive impacts on society through services that utilize natural language processing, computer vision, and speech recognition. However, it will also be important to improve GFSA by considering other dimensions of AI, such as robustness to adversarial examples, fairness, and explainability.

C Oversmoothing and Additional Visualizations

In Fig. 2, we show the visualizations of oversmoothing characteristics in DeiT. We also provide visualizations in other domains. We show the filter response, cosine similarity, and singular value of BERT finetuned on STS-B dataset of GLUE tasks in Fig. 5 and Graphormer finetuned on ZINC dataset in Fig. 6.

To characterize self-attention, we first analyze the filter response of self-attention in the frequency domain. We follow the method used by Wang et al. [80] for spectral visualization of the self-attention matrix. As shown in Fig. 2 (a), DeiT has a near-zero magnitude for the high frequencies, which is characteristic of a low-frequency filter and is likely to result in oversmoothing when applied multiple times.

We follow the calculation method of Guo et al. [28] for cosine similarity. As shown in Fig. 2 (b), the higher similarity as the layers of the model get deeper is related to the oversmoothing problem. To further analyze this issue, we also consider the dimensionality collapse in Transformer-based models. We plot the singular value distribution of the feature in the last block. As shown in Fig. 2 (c), insignificant, near-zero values dominate the feature distribution. As layers get deeper, the similarity of features increases and dimensional collapse occurs. The oversmoothing problem is the same in BERT and Graphormer, as shown in Fig. 5 and Fig. 6.

-40

-20

0

20

40

Frequency

0.0

0.2

0.4

0.6

0.8

1.0

Normalized Magnitude

BERT

BERT + GFSA

(a) Filter reponse

1

4

8

12

Layer Index

0.4

0.6

0.8

Cosine Similarity

BERT

BERT + GFSA

(b) Cosine Similarity

0

10

20

30

40

50

60

Singular Value Index

0.0

0.3

0.6

0.9

Normalized Singular Value

BERT

BERT + GFSA

(c) Singular Value

0

5

10

15

Frequency

0.0

0.2

0.4

0.6

0.8

1.0

Normalized Magnitude

Graphormer

Graphormer + GFSA

(a) Filter reponse

1

4

8

12

Layer Index

0.0

0.2

0.4

0.6

0.8

Cosine Similarity

Graphormer

Graphormer + GFSA

(b) Cosine Similarity

0

2

4

6

8

Singular Value Index

0.0

0.3

0.6

0.9

Normalized Singular Value

Graphormer

Graphormer + GFSA

(c) Singular Value

D Analysis of Frequency Responses with Visualizations

We analyze the frequency responses, which represent the impact of learned coefficients, for all 12 layers of BERTBASE with and without GFSA. From Fig. 7, our analysis reveals that GFSA learns various filter types between layers. In early layers, we observe a tendency towards low-pass filtering, with prominent peaks at low frequencies. This aligns with the need for broader feature extraction in initial layers. The middle layers show a mix of low-pass and high-pass characteristics, with more complex frequency responses. This suggests GFSA is learning to balance between feature extraction and refinement. In deep layers, there is a noticeable shift towards higher frequency responses, indicating a move towards high-pass filtering. This shift supports our claim that GFSA can mitigate oversmoothing in deeper layers. BERTBASE+GFSA shows a consistently higher magnitude response at higher frequencies, especially in deeper layers, compared to vanilla BERT. In other word, vanilla self-attention works primarily as a low-pass filter, while GFSA utilizes a wider range of frequencies.

-40

-20

0

20

40

0.0

0.2

0.4

0.6

0.8

1.0

BERT

BERT + GFSA

-40

-20

0

20

40

0.0

0.2

0.4

0.6

0.8

1.0

-40

-20

0

20

40

0.0

0.2

0.4

0.6

0.8

1.0

-40

-20

0

20

40

0.0

0.2

0.4

0.6

0.8

1.0

-40

-20

0

20

40

0.0

0.2

0.4

0.6

0.8

1.0

-40

-20

0

20

40

0.0

0.2

0.4

0.6

0.8

1.0

-40

-20

0

20

40

0.0

0.2

0.4

0.6

0.8

1.0

-40

-20

0

20

40

0.0

0.2

0.4

0.6

0.8

1.0

-40

-20

0

20

40

0.0

0.2

0.4

0.6

0.8

1.0

-40

-20

0

20

40

0.0

0.2

0.4

0.6

0.8

1.0

-40

-20

0

20

40

0.0

0.2

0.4

0.6

0.8

1.0

-40

-20

0

20

40

0.0

0.2

0.4

0.6

0.8

1.0

Frequency

Normalized Magnitude

E Frequency Analyses in the Singular Value Domain

Graph signal processing (GSP) [67, 68] can be understood as a generalized concept of DSP - in other words, DSP is a special case of GSP where a line graph with n nodes is used and therefore, the graph Fourier transform (GFT) of the line graph is identical to the discrete Fourier transform.

In the definition of GFT, we assume that the graph shift operator (GSO) S is diagonalizable. Considering the eigendecomposition of the GSO S = V ⊺ Λ V with eigenvector V , we can write the graph filter output as follows:

where x ∈ R n is a 1-dimensional graph signal, Λ is a diagonal matrix with eigenvalues, and w k ∈ [ -∞ , ∞ ] is a coefficient.

However, one can use the singular value decomposition, when the GSO is not diagonalizable but symmetrically normalized, instead of the eigendecomposition [49]. Both the singular value decomposition and the eigendecomposition project the original signal onto a set of basis, but they use different basis sets. In the singular value decomposition, we sort the set of basis in ascending order of their eigenvalues, and perform frequency domain-like analyses [106, 49].

Since the self-attention matrix's row-wise sum is always 1, the following is the case: ¯ A = D -1 A = 1 n A , where n is the number of tokens. Maskey et al. [49] define the following symmetrically normalized adjacency matrix (SNA): D -1 / 2 in AD -1 / 2 out . Since the degree of every node is n in the self-attention matrix, the following is the case: D -1 / 2 in AD -1 / 2 out = D -1 / 2 AD -1 / 2 = 1 √ n A 1 √ n =

1 n A = ¯ A . Therefore, the self-attention matrix is a special case of SNAs.

F Matrix Polynomial vs. Graph Fourier Transform

There are two paradigms of implementing graph filters: i) matrix polynomial, which does not require diagonalizability, and ii) graph Fourier transform, which uses the eigendecomposition for diagonalizable adjacency matrices or uses the Jordan decomposition or the singular value decomposition for non-diagonalizable adjacency matrices.

Those two paradigms have their own weaknesses: i) the matrix polynomial approach requires explicit matrix multiplications, and ii) the graph Fourier transform approach requires expansive spectral decompositions. The matrix polynomial is preferred when there are not many matrix multiplications. Otherwise, the graph Fourier transform approach may be better since the matrix multiplication can be simplified after the decomposition.

Among those two, we use the first matrix polynomial approach with only three non-zero coefficients { w 0 , w 1 , w K } since it does not require the complicated spectral decomposition. Since we do not rely on any explicit spectral decomposition but on the matrix polynomial, any adjacency matrix can be used.

G Proof of Theorem 3.1

Theorem 3.1 (Filter characteristics based on coefficient values) . Let ¯ A be a self-attention matrix interpreted as a graph with connected components. Consider the polynomial graph filter defined by ∑ K k =0 w k ¯ A k , where w 2 , w 3 , . . . , w K -1 = 0 and only w 0 , w 1 , and w K are non-zero. If the coefficients w k for k = 0 , 1 , K are positive and their sum is 1, then the polynomial filter acts as a low-pass filter, attenuating high-frequency components and promoting smoothness across the graph. Conversely, if w k = ( -α ) k for k = 0 , 1 , K and α ∈ (0 , 1) with sufficient large K , the polynomial filter exhibits high-pass filter behavior.

Note that without filtering, the singular value ratio is ∣ ∣ σ 0 i ∣ ∣ / ∣ ∣ σ 0 1 ∣ ∣ = 1 . In the case where | g ( σ i ) /g ( σ 1 ) | < 1 ∀ i ≥ 2 , it implies that after applying the graph filter g , the lowest frequency component further dominates, indicating that the graph filter acts as a low-pass filter. Conversely, in

the case where | g ( σ i ) /g ( σ 1 ) | > 1 ∀ i ≥ 2 , it implies that after applying the graph filter g , the lowest frequency component σ i no longer dominates, indicating that the graph filter acts as a high-pass filter.

Proof. We prove the low-pass filter result. For the case where w 0 , w 1 , and w K are positive and their sum is 1 , we show that

Hence, proving Theorem G is equivalent to show | g ( σ i ) | < 1 .

For the high-pass filter result, when w k = ( -α ) k / ( k +1) where k = 0 , 1 , K and α ∈ (0 , 1) , then we show that when w 0 = 1 , w 1 = -α/ 2 ,

since

It shows that the graph filter with w k = ( -α ) k / ( k +1) for k = 1 , 2 , K emphasizes high-frequency components and acts as a high-pass filter.

This proof supports that the behavior of the polynomial filter as either a low-pass or high-pass filter directly depends on the sign and values of the coefficients, as specified in Theorem 3.1.

H Proof of Theorem 4.1

Proof. The Frobeinus norm of the self-attention is directly related to how far the softmax probabilities are from being uniform. For any matrix M ∈ R m × n , we have

where S i is the i -th row of softmax ( M ) , U n is the uniform distribution over n elements, and d χ 2 ( p, q ) = ∑ i q i ( p i /q i -1) 2 is the χ 2 -divergence between p and q [14]. The Frobenius norm is maximized when the whole mass of the probabilities is on one element, which is a case for d χ 2 ( S i , U n ) = n -1 and ∥ softmax ( M ) ∥ F = √ m . Therefore, we can calculate the upper bound of Frobenius norm for ¯ A as follows:

Note that ¯ A ∈ R n × n is a right-stochastic matrix normalized with row-wise softmax: i) all the elements of ¯ A lie within [0, 1], and ii) the row-sum in ¯ A is equal to 1. Since the self-attention matrix

is a right-stochastic matrix, the power of the self-attention is also a right-stochastic matrix. Therefore, Eq. (21) is also hold for ¯ A K as follows:

Now, considering the error term E K as given by Theorem 4.1, and applying the triangle inequality for matrix norms:

I Implementation of GFSA

The pseudo code of our GFSA is shown in Algorithm 1. For implementation, w 0 and w 1 can be set as hyperparameters optionally.

Algorithm 1 PyTorch-style pseudocode for GFSA

J Comparison with Actual and Approximated High-order Terms

To compare the impact of the actual ¯ A K and the approximated ¯ A K in terms of accuracy, we experimented with BERT on GLUE and the results are summarized in Table 9. BERTBASE+ ¯ A K denotes using the exactly calculated ¯ A K instead of the approximated ¯ A K .

K Natural Language Understanding

K.1 Detailed Experimental Settings

We integrate GFSA into 3 pre-trained large language models: BERT, ALBERT, and RoBERTa. We evaluate them on the GLUE benchmark, which includes 3 categories of natural language understanding tasks: i) single-sentence tasks CoLA and SST-2; ii) similarity and paraphrasing tasks MRPC, QQP, and STS-B; iii) natural language inference tasks MNLI, QNLI, and RTE. For MNLI task, we experiment on both the matched (MNLI-m) and mismatched (MNLI-mm) versions. Following Devlin et al. [16], we report Matthews correlation for CoLA, F1 scores for QQP and MRPC, Spearman correlations for STS-B, and accuracy scores for the other tasks. For each task, we select the best hyperparameters for GFSA, and the other hyperparameters are fixed. We compare our GFSA with ContraNorm [28], one of the related methods that address oversmoothing. We finetune ContraNorm with the recommended hyperparameters in Guo et al. [28]. We initialize with a pre-trained language model and finetune with added GFSA for 5 epochs.

Dataset. The benchmark datasets we used are listed below.

CoLA. The Corpus of Linguistic Acceptability [85] consists of English acceptability judgments drawn from books and journal articles. The target task is a binary classification task, and each sentence is determined to be grammatically acceptable or not.

SST-2. The Stanford Sentiment Treebank [73] is a dataset in which each sentence is sourced from movie reviews and accompanied by human annotations of their sentiment. The target task is to classify binary sentiments for a single sentence.

MRPC. The Microsoft Research Paraphrase Corpus [17] is a corpus of sentence pairs, which are automatically extracted from online news sources and annotated by humans. The target is to determine whether the sentences in the pair are semantically equivalent.

QQP. The Quora Question Pairs [11] dataset is a collection of question pairs from the community question-answering website Quora. The target is to determine whether the questions in the pair are semantically equivalent.

STS-B. The Semantic Textual Similarity Benchmark [8] is a collection of sentence pairs drawn from news headlines, video and image captions, and natural language inference data with human annotation. The target is a regression task to predict a similarity score from 0 to 5.

MNLI. The Multi-Genre Natural Language Inference Corpus [87] is a crowdsourced collection of sentence pairs with textual entailment annotations. Given a premise sentence and a hypothesis sentence, the task is to predict whether the premise entails the hypothesis (entailment), contradicts the hypothesis (contradiction), or neither (neutral). The standard test set consists of private labels from the authors and evaluates both the matched (in-domain) and mismatched (cross-domain) sections.

QNLI. The Stanford Question Answering [82] dataset is a question-answering dataset consisting of question-paragraph pairs, where one of the sentences in the paragraph contains the answer to the corresponding question written by an annotator. The task is to determine whether the context sentence contains the answer to the question.

RTE. The Recognizing Textual Entailment [5] dataset comes from a series of annual textual entailment challenges. The target task is a binary entailment classification task.

BERT. BERT [16] consists with 12 layers, 12 heads, 768 hidden size, 512 maximum sequence length, and MLP dimension of 3072.

ALBERT. ALBERT [40] consists of 12 layers, 12 heads, 768 hidden dimensions, 512 maximum sequence length, 128 embedding dimensions, and MLP dimension of 3072.

RoBERTa. RoBERTa [44] consists of 12 layers, 12 heads, 768 hidden size, 514 maximum sequence length, and MLP dimension of 3072.

Training. For implementation, we adopt HuggingFace framework. We trained all models with 5 epochs with 32 batch size. The linear learning rate decay is used and initial learning rate is set to 2 × 10 -5 . We use AdamW [46] optimizer, and weight decay is set to 0. All models are trained on 1 GPU and of NVIDIA RTX A5000 24GB.

K.2 Sensitivity to K

In this section, we explore the influence of the polynomial order, denoted as K , in our GFSA, conducting experiments on BERTBASE finetuned with GLUE tasks. We search for values of K from 2 to 10, and the results are presented in Table 10. For each dataset, there is an optimal K and the performance of models using GFSA is generally robust to changes in K .

L Causal Language Modeling

L.1 Detailed Experimental Settings

Dataset. The benchmark datasets we used are listed below.

PTB. Penn Treebank [47] dataset is a collection of text documents that have been extensively annotated with linguistic information, primarily syntactic and grammatical structures.

WikiText. WikiText [50] dataset is a collection of over 100 million tokens extracted from the set of verified good and featured articles on Wikipedia. Compared to the preprocessed version of Penn Treebank (PTB), WikiText-2 is over 2 times larger and WikiText-103 is over 110 times larger.

GPT2. GPT2 [61] is a Transformer pretrained on a very large corpus of English data in a selfsupervised fashion without any human labelling on dataset. It automatically generate inputs and labels from those texts, and trained to guess the next word in sentences. For implementation, we adopt HuggingFace Framework 4 . For all experiments, GPT2 has 12 layers with 12 attention heads, 768 hidden size and 1024 maximum sequence length, resulting in a total of 117 million parameters.

Training. We finetune GPT2 with 4 batch size, 5 × 10 -5 learning rate and linear learning weight decay using adamW [46] optimizer. We also apply dropout with probability 0.1. Following [93], we train models for 15 epochs with PTB, 4 epochs with WikiText-103 and 10 epochs with WikiText-2. We use sensitivity metric, i.e., perplexity, which is a commonly used metric to evaluate the performance of language models, particularly in language modeling and text generation tasks. perplexity measures how well a language modeling can predict a sequence of words in a given text or a test dataset. All the experiments are conducted on 1 GPU and of NVIDIA RTX 3090 24GB.

L.2 Sensitivity to K

We conducted a sensitivity study on K of GPT-2 across all datasets, and the results are presented in Table 11. For PTB and WikiText-2, GFSA exhibits the best performance when K is high, typically

around 8 or 9. However, for WikiText-103, GFSA achieves the best perplexity when K is small, specifically when K is 3 or 4.

M Image Classification

M.1 Detailed Experimental Settings

Our code is implemented based on the timm library [86]. In the case of our training recipe, it is the same as experimental setting of Wang et al. [80] that follows the training recipes of Touvron et al. [74] and Touvron et al. [75]. To apply our GFSA to existing base models such as DeiT, Cait, and Swin, we consider a range of K between 2 and 5. For 12-layer DeiT, we follow the same hyperparameters from Wang et al. [80]. We set the dropout rate to 0 and 0.2 for 12-layer and 24-layer DeiT, respectively. For CaiT, we apply our GFSA on only to the patch embedding layer. All other hyper-parameters are kept consistent with the original papers of DeiT [74], CaiT [75] and, Swin [45]. All models are trained on NVIDIA RTX 3090 24GB.

M.2 FLOPs & Throughput

In Table 12, we report the number of FLOPs and throughput. With GFSA plugged in, the FLOP count is either the same or no different. For DeiT-S with 24 layers, which shows a slight FLOP increase with GFSA plugged in. However, for the rest of the settings, the models have the same number of Flops. For throughput, it tends to decrease because calculating the high-order term is an additional cost.

M.3 Sensitivity to K

We also perform the sensitivity analysis for K . Tables 13 and 14 show the results of sensitivity analysis for DeiT-S and CaiT-S with GFSA plugged in. For 12-layer DeiT-S, GFSA performance of 81.12 is highest when K = 3 . When the GFSA has a K of 2, the performance is worse than the

original DeiT-S, but when the K is 3 or higher, the performance is better than the original DeiT-S, and most surprisingly, the performance is better than the 24-layer DeiT-S.

CaiT-S shows the highest performance of 82.84 when K = 4 . For CaiT-S, the accuracy is slightly lower than that of the original CaiT-S when K = 2 , but it starts to exceed the accuracy of CaiT-S when K is 3 or higher.

M.4 Full Experimental Results

In Table 15, we consider all three classes, CNN only, CNN + Transformer, and pure Transformer, to compare more different models than in Table 3. In particular, in the Transformer category, we only test with lightweight models with similar number of parameters, such as ViT-S and DeiTS. Compared to existing techniques, the improvements by GFSA already surpasses LayerScale (0.7%) [75], LateInsertion (0.6%) [75], and HAT [4] (1.38%).

M.5 Additional Comparison with SOTA Models

Our main experiment aims to determine whether introducing the GFSA layer would help improve performance in a base model, such as DeiT. We also compare our method with the recent models: SpectFormer [58], SVT [57], NeuTRENO [52], FNet [42], and GFNet [64]. We use a 12-layer setup to ensure a fair comparison in Table 16.

GFNet [64] can reduce the number of parameters, but there is a performance penalty. However, the performance improvement of DeiT-S+GFSA is relatively greater than DeiT-S compared to other models. SpectFormer [58] and SVT [57] have advantages in calculation amount and model complexity, and performance is improved over DeiT-S, but Top-1 and Top-5 accuracies are lower than those using GFSA. Additionally, NeuTRENO [52] also improves as much as GFSA compared to DeiT-S, but GFSA still has higher Top-1 accuracy.

M.6 Additional Experiments with Guo et al. [28]'s setting

To make a fair comparison with ContraNorm [28], one of the related studies that mitigates oversmoothing, we run additional experiments to match their experimental setup.

Setting. We follow the training recipe used by Guo et al. [28], which is a slightly modified version of Touvron et al. [74]'s recipe. Guo et al. [28] use AdamW optimizer with cosine learning rate decay. We select the DeiT-T and DeiT-S for ImageNet-1k. 'T' and 'S' denote tiny and small model sizes, respectively. For all experiments, the image size is set to be 224x224. We train each model for 300 epochs and the batch size is set to 1024. For ContraNorm, we train with their recommended hyperparameters. All models are trained on 4 GPUs and of NVIDIA RTX A6000 48GB.

Results. In Table 17, DeiT-T and DeiT-S with GFSA outperform vanilla DeiT-T and DeiT-S in all layer settings. GFSA improves the performance of DeiT-T with 12 layers by 1.52%. The largest gain is a 4.88% improvement on 16-layer DeiT-T. This shows that the effect of GFSA is larger than the effect of ContraNorm. For DeiT-S with 16 layers, surprisingly, GFSA is able to increase the performance by 80.83%, meaning that GFSA brings benefits with a 3.23% improvement.

Sensitivity to K . In Table 18, we experiment with a sensitivity analysis for K . For DeiT-T, the performance of GFSA generally improves when K is 4 or 5. On the other hand, GFSA performs better at lower K for settings that are layers 16 and 24 for DeiT-S.

N Automatic Speech Recognition

N.1 Detailed Experimental Settings

Dataset. We conduct experiments on the LibriSpeech 5 dataset [55], which consists of audio recordings paired with their transcriptions. The LibriSpeech dataset has approximately 1,000 hours of read English speech with a sampling rate of 16 kHz. We keep the original 16,000Hz sampling rate and compute 80-dim log-Mel filterbanks for a 25ms sliding window, strided by 10ms. The filterbank features are then normalized to zero mean and unit variance per input sequence. For implementation, we follow the recipes of SpeechBrain [65].

Evaluation Metric. Word error rate (WER (%)) is derived from the Levenshtein distance and compares a reference to a hypothesized word-level transcription. It is calculated by summing the number of word insertions, deletions, substitutions and dividing it by the total number of words in the reference transcription.

Vanilla Transformer. We use a vanilla Transformer to apply our GFSA. For implementation, we use a SpeechBrain [65] framework. The vanilla Transformer consists of i) 1D convolution to perform striding, ii) Transformer encoder with 12 layers, 4 heads, embedding dimension of 512, MLP dimension of 2048, and post-LayerNorm iii) decoder with 6 layers, 4 heads, embedding dimension of 512, MLP dimension of 2048, joint beamsearch, and iv) external Transformer language model with 12 layers, 12 heads, embedding dimension of 768, and MLP dimension of 3072.

Branchformer. We use one of the SOTA models, Branchformer [59] to plug-in our GFSA. Branchformer has two parallel branches, one for capturing global interactions using attention and the other for more localized context using convolutional gating MLP. The Branchformer architecture for speech recognition consists of i) 1D convolution to perform striding, ii) Branchformer encoder with 18 layers, 8 heads, embedding dimension of 512, and MLP dimension of 3072, iii) decoder with 6 layers, 8 heads, embedding dimension of 512, a convolutional spatial gating unit (CSGU) dimension of 3072, joint beamsearch, and iv) external Transformer language model with 12 layers, 12 heads, embedding dimension of 768, and MLP dimension of 3072.

Training. We follow a training recipe from SpeechBrain [65]. The standard LibriSpeech validation sets (dev-clean and dev-other) are used to tune all parameters and select the best models. Test sets (test-clean and test-other) are used only to report final WER performance. We train the pure Transformer for 100 epochs and the Branchformer for 120 epochs with a batch size of 16. We use a data augmentation method on all models using SpecAugment [56]. SpecAugment applies time and frequency masking as well as time warping to the input spectrum. For Branchformer, we use AdamW [46] optimizer with 0.9 and 0.98 coefficients for computing running averages of gradient and its square. The learning rate and weight decay in all models are 0.0008 and 0.01, respectively. We use a connectionist temporal classification (CTC) loss [26, 100]. We also apply dropout with probability 0.1 and label smoothing with weight 0.1 to mitigate overfitting. We fix the random seed as 74443 on all experiments. All models are trained on 1 GPU and of NVIDIA RTX A6000 48GB.

Hyperparameters. In Table 19, we describe main hyperparameters used in the automatic speech recognition task. For Transformer+GFSA and Branchformer+GFSA, we also report the best K hyperparameter.

N.2 Training Curve

We compare the training and validation curves for LibriSpeech 100h in Fig. 8. The training loss curve of GFSA is lower than the pure Transformer. GFSA stabilizes the loss curve of pure Transformer slightly earlier.

0

25

50

75

100

Epoch

0

5

10

15

20

Train Loss

Transformer

Transformer+GFSA

(a) Training loss curve

0

25

50

75

100

Epoch

0

5

10

15

20

25

30

Transformer

Transformer+GFSA

(b) Validation loss curve

O Graph-level Tasks

O.1 Detailed Experimental Settings

Experimental settings for Graphormer. We describe benchmark datasets and Graphormer as the backbone model we used. ZINC [35] is the most popular real-world molecular dataset to predict graph property regression for constrained solubility, an important chemical property for designing generative GNNs for molecules. Uniform sampling is adopted for data splitting. We use a ZINCsubset of small-scale dataset. PCQM4M-LSC [33] is 2D molecular graphs, which is one of the most practically relevant quantum chemical properties of molecule science. The task is to predict density functional theory (DFT)-calculated HOMO-LUMO energy gap of molecules given their graphs. PCQM4M-LSC is unprecedentedly large in scale comparing to other labeled graph-level prediction datasets, which contain more than 3.8M graphs. We use PCQM4M and PCQM4Mv2 large-scale datasets.

Following Graphormer [94], we use Graphormer for PCQM4M and GraphormerSLIM for ZINC. Graphormer consists of 12 encoder layers, 80 encoder embedding dimension, and 768 MLP dimension. It employs 32 encoder heads and 24 hidden dimension for each head. GraphormerSLIM consists of 12 encoder layers, 80 encoder embedding dimension, and 80 MLP dimension. It employs 8 encoder heads and 10 hidden dimension for each head. We use adamW [46] optimizer with 0.9 and 0.999 coefficients for running averages of gradient and its square, and use Mean Absolute Error (MAE) as loss function. We use polynomial learning rate decay, with initial learning rate set to 2 × 10 -4 and end learning rate set to 1 × 10 -9 . For ZINC, we set batch size as 256, max epochs as 10k, and warm-up stage step as 40k. For PCQM4M and PCQM4Mv2, we set batch size as 1024, max epochs as 300, and warm-up stage step as 60k. All models are trained on 1 GPU and of NVIDIA RTX 3090 24GB. We conduct experiments with 4 different seeds.

Experimental settings for GPS and Graph-ViT. Weuse various benchmark datasets to experiment with our GFSA on GPS [63] and Graph-ViT [30]: Peptide-func and Peptide-struct from Long-Range Graph Benchmark (LRGB) [21], MNIST, CIFAR10, and ZINC from Benchmarking GNNs [22], and Moltox21 and Molhiv from OGB [32]. We fix all hyperparameters of GPS and Graph-ViT as recommended in their paper to ensure a fair comparison. To plug GFSA into GPS, we replace the self-attention module of GPS with GFSA. For Graph-ViT, we apply GFSA instead of the Hadamard self-attention mechanism and compare it with this self-attention method. We conduct experiments on GPS and Graph-ViT in their open code frameworks for a fair comparison:

GPS: https://github.com/rampasek/GraphGPS

Graph-ViT: https://github.com/XiaoxinHe/Graph-ViT-MLPMixer

O.2 Experimental Results with Standard Deviation

We conduct experiments following the experimental environments of Graphormer [94] using 4 different seeds. Due to space constraints, only the mean values are reported in Tables 4 and 5. In Tables 20 and 21, we report the results with mean and standard deviations.

P Code Defect Detection

P.1 Detailed Experimental Settings

Dataset. We use Devign dataset provided by [105], which is a binary classification task to evaluate whether a C language code is vulnerable to software systems or not.

Implementation. We build our experiments on top of the open-sourced code 6 and recipes provided by Wang et al. [84].

RoBERTa. RoBERTa [44] is an encoder-only model trained with masked language modeling on code. All hyperparameters are consistent with the training method in the source code of Wang et al. [84].

PLBART. PLBART [2] is an encoder-decoder model based on BART [43] architecture. PLBART can support understanding and generation tasks. All hyperparameters are consistent with the training method in the source code of Wang et al. [84].

CodeBert. CodeBERT [23] is a model trained on masked language modeling and replaced token detection. CodeBERT is a bimodal pretrained model based on Transformer with 12 layers for programming language and natural language. All hyperparameters are consistent with the training method in the source code of Wang et al. [84].

CodeT5. CodeT5 is an encoder-decoder framework with the same architecture as T5 [62]. It aims to derive generic representations for programming language and natural language via pre-training on unlabeled source code. CodeT5-small has 6 encoder layers, 6 decoder layers, 8 attention heads, 512 dimensional hidden states, and 60M parameters. The other models have 12 encoder layers, 12 decoder layers, 12 attention heads, 768 dimensional hidden states, and 220M parameters. All hyperparameters are consistent with the training method in the source code of Wang et al. [84].

Training. The pre-trained models mentioned above are applied to this downstream task. We add GFSA directly on top of self-attention. We finetune baselines and GFSA models for 10 epochs with a batch size of 16. We use early stopping strategy with a patience of 2. Models generate binary labels from unigram sequences at the decoder for defect detection task. We employ accuracy for evaluating the code defect detection task. All models are trained on 1 GPU and of NVIDIA RTX A6000 48GB.

P.2 Case Study

In Listing 1, we show one of case for code snippets of defects in QEMU 7 that CodeT5-base does not predict correctly, but that only CodeT5-base+GFSA predicts. The commit message 8 for this case is as follow:

Needed for changing cpu_has_work() argument type to CPUState, used in h_cede() .

h_cede() is the hypercall that asks the hypervisor to shut down the CPU. Previously, this hypercall simply passed the CPUID, so the hypervisor did not know what state the CPU was in. This change allows the hypervisor to know whether the CPU is actually performing work. If the CPU is performing a task, the hypervisor waits for the CPU to complete the task.

In this context, accurately predicting defects like the one above is very important, and applying GFSA to CodeT5-base helps in terms of performance improvement.

Q Code Clone Detection

Q.1 Detailed Experimental Settings

Dataset. Code clone detection aims to measure the similarity between two code snippets and predict whether they have the same functionality. We experiment with the Java data provided by Wang et al. [83].

Implementation. We build our experiments on top of the open-sourced code 9 and recipes provided by Wang et al. [84].

Training. We finetune both CodeT5 and CodeT5+GFSA for one epoch with a batch size of 16. We also use early stopping with patience of 2. CodeT5 and CodeT5+GFSA encode source code and take the representation to calculate similarity of two code snippets. We employ F1 score for evaluating this task. All models are trained on 1 GPU and of NVIDIA RTX A6000 48GB.

Q.2 Experiment Result

Table 22 shows results comparing CodeT5 and CodeT5 with GFSA. The result shows that by using our GFSA, CodeT5 models improve their performance. CodeT5-small+GFSA provides a 0.61% improvment over Code5T-small.

R Time Complexity and Empirical Runtime Analysis

Time Complexity. The time complexity of original self-attention is O ( n 2 d ) . But our GFSA has a high order term. Therefore, the time complexity of GFSA has O ( n 2 d + n 3 ) . If n is smaller than d , the time complexity approaches O ( n 2 d ) , which is the complexity of original self-attention.

Empirical Runtime Analysis. We report the training time of various methods with GFSA in Tables 23 to 28. In general, the training time of methods with GFSA is slightly longer than that of existing methods. For example, the Transformer for the automatic speech recognition task increases from 190.5 seconds to 191.6 seconds on Librispeech 100h dataset, as increases of only 1 second. Instead of computing higher-order polynomial terms, our GFSA approximates them, with only a small increase in runtime, which is not very significant.

S Inference Time Analysis

We report the inference time of various methods with GFSA in Tables 29 to 34.

T Results of the Strategy for Efficiency

In Tables 35 to 39, the results show that this strategy can reduce the increase in runtime and maintain performance compared to using GFSA for all layers.

U GFSA in Linear Transformers

Table 40 shows the accuracy, runtime, and GPU usage results on Long Range Arena benchmark using ListOps and Image datasets.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly stated our claims in the introduction, including the contributions of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Section 6, we stated the computational overhead as a limitation of our method and provided strategies to address it.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have included proofs for Theorem 4.1 and Theorem 3.1 in Appendix H and Appendix G.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our reproducibility statements are reported in A.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See the statement about reproducibility in Appendix A.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental environments, datasets, training methods are reported in Section 5.1 to 5.6 and in Appendix K to 5.6.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We tried to follow the experimental setup for benchmarking each task. Among the tasks we considered, it is conventional to report results from four experiments in the graph-level tasks task, so we report the mean and standard deviation together in the Appendix O.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Our paper specifies the type of computer resource information (e.g. GPU) for the experiment. We also indicated the amount of calculation or execution time for the experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the potential positive and negative social impacts of our research in Appendix B.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There is no such risk in our paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: In our paper, the authors or original owners of code, data, and models used are properly credited, and their licenses and terms of use are explicitly mentioned.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code we provided is well documented in the README and is anonymized. Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our research does not include crowdsourcing experiments and studies with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Because our research does not involve human subjects, it is not subject to IRB approval.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo overcome the oversmoothing effects in general transformer settings, the author considers to treat the attention module as the graph filter and propose to use a polynomial graph filter (graph signal processing) techniques to alleviate oversmoothing. Specifically,  the author considers the induced attention matrix from features as the adjacency matrix and construct a filter that allows exhibiting high-pass and low-pass filter. As low-pass filters provides smoothing tendency and high-pass filter provides edge tendency, the author leverages an adaptive filter to alleviate oversmoothing effects on Transformer domain.\n\nSTRENGTHS:\n1. The idea is clear and well understood, the author leverages the idea in GNN fields to the transformer models.\n2. The high-pass and low-pass filter is analyzed and the author used an approximation to approximate the polynomial filter.\n3. The experiments covers many different applications of transformers.\n\nWEAKNESSES:\n1. The novelty is very limited. The idea of building adaptive high-pass, low-pass filter is well studied in GNN field and has been leveraged in many works to solve oversmoothing in GNN. The idea of attention module as graph filter is also proposed in many Graph transformer models. The main contribution of this paper is simply change the direction from using attention as graph filters to use graph filters as attentions, which is not very interesting.\n\nQUESTIONS:\nIn the Theorem F.1, the description is not very clear. I assume  \"the sum w0 +w1σi +wkσi^{k}  will be smaller than g(σi)\" indicates that g(σi) will become smaller as i increase, but this depends on the scale of w1 and wk, even as σi^{k}->0 with i is large, if the rate wk/w1 is large, such tendency can only be guaranteed asymptotically, in the limited i case, this doesn't always hold true. Please correct me if I am wrong.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel approach to enhance the self-attention mechanism in transformers by drawing on graph signal processing principles. The authors reframe the standard self-attention as a low-pass graph filter and introduce a more generalized filter (GFSA) capable of capturing low-pass, high-pass, or combined filtering behaviors. They evaluate the effectiveness of GFSA across diverse tasks, showcasing its potential to improve performance.\n\nSTRENGTHS:\n- The core idea is easily grasped and demonstrates promising results across various transformer variants in different domains.\n- The authors evaluate GFSA on a wide range of tasks spanning natural language processing, computer vision, speech recognition, and graph regression, highlighting its versatility.\n\nWEAKNESSES:\n- While the oversmoothing problem motivates the proposed method, the experiments lack a focused analysis of its impact on transformers with varying depths (number of layers).\n- While the paper conducted experiments in a variety of tasks in different fields to show the versatility of the proposed method, not study in depth each task could also be a potential weakness. In particular for the graph regression task, the authors only considered two datasets and a single baseline model (and the improvement over the larger model on PCQMv2 is marginal 0.860 vs 0.862). Stronger baselines such as [GPS](https://arxiv.org/abs/2205.12454) and additional datasets would provide a more rigorous assessment of GFSA's practical value. Additionally, the choice of the baselines in other tasks might also suffer from the same weakness, as I believe most of them are relatively old methods. Finally, their performance statement in Fig.1 may be misleading to practitioners without acknowledging the limitations with their selected baselines.\n- Given the abundance of research on oversmoothing in transformers, the paper should include comparisons with more recent and relevant methods (e.g., [64]) to strengthen its claims. Currently, the paper only compares to very few comparison partners (ContraNorm [27] and [73]) or even without any comparison in some tasks.\n- The paper states that the oversmoothing issue is due to the low-pass filter nature of the self-attention. By introducing a 2nd order term in Eq.10, they claim that their proposed method can learn both low-pass and high-pass filters, and even combined filters. To validate this empirically, a deeper analysis of the learned coefficients $w_0,w_1,w_K$ across different layers would provide concrete evidence for the GFSA's ability to learn diverse filter types.\n\nQUESTIONS:\n- L205: for the self-attention with the skip connection, shouldn't $w_0$ be positive?\n- In the NLP experiments, did the pretrained models use the standard self-attention? If so, does this approach make sense to replace them with GFSA only during fine-tuning?\n- The paper only discusses the runtime overheads. Does GFSA introduce any potential memory overheads?\n\nLIMITATIONS:\nThe paper discussed the additional runtime overhead of GFSA compared to the original self-attention. However, other potential limitations are not discussed, such as potential memory overheads, scalability of GFSA to long sequences, or extendability to efficient/linear transformers.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-11"}

comments: SUMMARY:
This work proposes to enhance self-attention by considering the high-order power of the attention matrix inspired by graph convolution networks and graph signal processing, to overcome the oversmoothing issues of transformers.
To reduce the computational overhead, the authors propose to use a first-order Taylor approximation to the higher-order power of the attention matrix.
With the slightly larger complexity, the proposed technique can improve the performance of Transformers in various fields, across computer vision, natural language processing, etc.

STRENGTHS:
1. This paper is well written.
2. The comprehensive empirical experiments well support the advantage of the proposed method and the statement of the work.

WEAKNESSES:
1. The notation in the equation on Line 99 is confusing (the "exponential" in the self-attention matrix).
2. The Taylor approximation well actually hurt the capacity of the graph filters significantly, degenerating to the power of 2, regardless of the value of $K$. 
The eq (10) to degenerate to $\hat{\omega}_0 \boldsymbol{I} + \hat{\omega}_1 \bar{\boldsymbol{A}} + \hat{\omega}_2 \bar{\boldsymbol{A}}^2$ for all $K$, where $\hat{\omega}_0, \hat{\omega}_1, \hat{\omega}_2$ are learnable coefficients. Therefore, this technique is just for the power of $2$ and the Taylor approximation actually does not contribute.

QUESTIONS:
1. Besides the runtime, can you also provide the GPU memory consumption comparisons between GFSAs and the base models?

LIMITATIONS:
No specific aware.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a graph-filter-based self-attention mechanism to improve its effectiveness. The authors conduct experiments in various fields, including natural language understanding, computer vision, automatic speech recognition, graph regression, and code classification, showing the generalization of the proposed method. The proposed method is claimed to make a difference in the over-smoothing problem, where representations across layers converge to indistinguishable values, leading to significant performance degradation.\n\nSTRENGTHS:\n1. Sufficient Experiments. The paper conducts experiments on natural language understanding, computer vision, automatic speech recognition, graph regression, and code classification. The authors try their best to show the generalization of the proposed method.\n2. Well Organized. The paper articulated the viewpoint in a clear and organized manner.\n3. Reproducibility. The paper shows the pseudo-code, which clearly presents the implementation of GFSA in the appendix. The paper shows detailed experiment settings in appendices I to O. The code is available with instructions. However, an error occurred when I tried to run it, which is reported in the following questions.\n\nWEAKNESSES:\n1. Sensitivity of K. The authors did not give instructions on selecting the value of K. The sensitivity of K is discussed in appendices, showing that 'For each dataset, there is optimal K and the performance of models using GFSA is generally robust to changes in K'. All the experiments searched for values of K. However, the search for K is time-consuming.\n2. Misleading Figure. Fig 1 shows the performance of the proposed method. However, the inconsistent proportions exaggerate the model's effectiveness.\n\nQUESTIONS:\n1. Miss code. When I was running the experiment of code classification, using the command “python run_exp.py --model_tag codet5_base --task defect --sub_task none” as instructed in the readme file, an error occurred reporting ‘can't open file '/run_gen.py'’ . It seems that some code is missing.\n2. More questions about the sensitivity of K. Is it a necessary step to search the optimal K when facing a new dataset? Or is it robust in K? I noticed that, in Theorem 4.1, Ek is relevant to K.\n\nLIMITATIONS:\nThe authors addressed the limitations, including effectiveness and efficiency.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}