Scalable Hierarchical Self-Attention with Learnable Hierarchy for Long-Range Interactions
Authors: Thuan Nguyen Anh Trang
Khang Nhat Ngo
Hugo Sonnery
Thieu Vo
Siamak Ravanbakhsh
Truong Son Hy
Venue: Accepted by TMLR ()

Abstract: Self-attention models have made great strides toward accurately modeling a wide array of data modalities, including, more recently, graph-structured data. This paper demonstrates that adaptive hierarchical attention can go a long way toward successfully applying transformers to graphs. Our proposed model Sequoia provides a powerful inductive bias towards long-range interaction modeling, leading to better generalization. We propose an end-to-end mechanism for a data-dependent construction of a hierarchy which in turn guides the self-attention mechanism. Using adaptive hierarchy provides a natural pathway toward sparse attention by constraining node-to-node interactions with the immediate family of each node in the hierarchy (e.g., parent, children, and siblings). This in turn dramatically reduces the computational complexity of a self-attention layer from quadratic to log-linear in terms of the input size while maintaining or sometimes even surpassing the standard transformer's ability to model long-range dependencies across the entire input. Experimentally, we report state-of-the-art performance on long-range graph benchmarks while remaining computationally efficient. Moving beyond graphs, we also display competitive performance on long-range sequence modeling, point-clouds classification, and segmentation when using a fixed hierarchy. Our source code is publicly available at https://github.com/HySonLab/HierAttention

Published in Transactions on Machine Learning Research (04/2024)
Scalable Hierarchical Self-Attention with Learnable Hierarchy for Long-Range Interactions
Thuan Trang" FPT Software Al Center
thuantna@@ssft.com..o
Nhat Khang Ngo*
khangrnn @8fpt. com
PPT Software AI Center
Hugo Sonnery*
husp.sonnery@@ail.mmpoi
McGill University
Mila Quebec Al Institute
Thicu N. Vo
wongorthiicu@dduu.ccu
Ton Duc Thang University
FPT Software Al Center
Siamak Ravanbakhsh
Baemak@@s..mgglllee
McGill University
Mila Quebee AI Instilute
Truong Son Hy
TruongSon. TruongSon.Hgiindstaon www.Hdyyttteeeeeedde
Indiana State University FPT Software Al Center
Reviewed on OpenReview: https: // optnr/viewwwww.. net/ forum? id= qH4 VFMyhce
Abstract
Self-attention models have made great strides toward accurately modeling a wide array of data modalities, including, more recently, graph-structured data. This paper demonstrates that adaptive hierarchical attention can go a long way toward successfully applying trans- formers to graphs. Our proposed model Sequoia provides powerful inductive bias towards long-range interaction modeling, leading to better generalization. We propose an end-to-end mechanism for a data-dependent construction of a hicrarchy which in turn guides the self- attention mechanism. Using adaptive hierarchy provides a natural pathway toward sparse attention by constraining node-to-node interactions with the immediate family each node the hierarchy (e.g., parent., children, and siblings). This turn dramatically reduces the computational complexity of oeff-attention layer from quadratic log-linear in terms of the input size while maintaining sometimes to model long-range dependencies across the entire input. Experimentally, we report state- of-thc-art performance on long-range graph benchmarks while remaining computationally efficient. Moving beyond graphs, we also display compelitive performance on long-range se- quence modeling, point-clouds classification, and segmentation when using a fixed hierarchy. Our source code publicly available at lhttp:: htaps:///ithub... http:/hhw.ccc/t/thooa////eeettte
surpassing the standard transformer's ability
"These authors contributed equally to this work *Correspondence Manonngon.Hy@inneeee

---

Published in Transactions on Machine Learning Research (04/2024)
1 Introduction
Transformers Vaswani et al. 2017 are powerful models that demonstrated superior performance in various input modalities, ranging from natural language processing to video understanding. Self-attention has also proved to be an effective tool for handling long-range dependencies, although its efficiency suffers due it quadratic complexity
the input size. Thercfore, considerable body of literature on efficient transformers is dedicated to scaling the vanilla transformer with multiple approaches ocaling/performante performance trade-ofT. Strategies to make them more compute-efficient include approximating the attention matrix with sparsity patterns (Ho et al. 2020 [Blltagy al. 2020a Zaheer et al 2020a),, clustering before computing attention (Roy et al. 2021 Kitaev ct al. 2020b). low-rank estimation
resolving their
[Wang et al. 2020:
Xiong et al. 2021 and better memory I/O (Do et al. 2022
Ancestor A A
Target Node
Siblings J
Children C2s
Figure 1: (Left) We adaptively partition the graph form a hierarchy. This incremental partitioning at cach level is produced by message passing the graph at lower level and the use of Gumbel-mat for hard clustering. (Right) This adaptive tree forms the basis for sparse self-attention in single self-attention layer.
In this work, we focus on devising a self-attention mechanism that could enable long-range interaction modeling on graph-structured data. Most current efficient transformers either sparsify the attention matrix using random patterns or distance metric dcrived from the tokons' relative positions in a sequence. Our initial motivation coincides with similar idea of skipping superfluous interactions to capture long- range interactions under fixed computational budget (Choromanski et al.) [202]] Katharopoulos et al., 2020). More precisely, model Sequoia decreases the computational cost by exploiting task-informed hierarchical clustering of the graph rather than using prespecified interaction pattern. As a result, our method not only requires lower resources but also surpasses the performance of standard attention due to additional information about the data structure provided by the learnable hierarchy. Furthermore, by propagating the information bottom-up and top-down through the tree built atop the input graph, our method guarantees local and global information sharing. The resulting adaptive hicrarchical self-attention layer can be used as building block for geometric deep lcarning using transformers.
adaptive
Our model is comprised of two parts (i) computational tree learning that forms the basis for (ii) hi- crarchical attention mechanism. Our work attempts to cover long-range datascts with various structures, including what we calegorize as ordered, semi-ordered, and unordered data. This distinction is related to the way the hierarchy is constructed each case: for ordered data structures, such as images and long soquences, hicrarchy may be naturally constructed from their regular grid. More concretely, a sequence or image is divided into nested sub-sequence or image patches, with decreasing resolution and increasing receptive field at each level of the hierarchy. For semi-ordered structures such point clouds, such grid
2

---

Published in Transactions on Machine Learning Research (04/2024)
structure can be "imposed", for example, using nested voxelization (Liu al. 2019) However, graphs, example of an unordered structure, arguably lack natural hierarchy, and we learn the hierarchy for each data instance. Due to the effectiveness of adaptive hierarchy for graphs, we focus our discussions on graphs; discussion and further results on long-range sequences and point clouds are moved to the appendix.
In short, the main contributions of this paper are:
An efficient hierarchical self-attention scheme with O(n log(n)) space and time complexity in terms of the sequence length the number of points / the graph size. A data-driven approach to constructing a tree as a guideline for the attention mechanism for long-range graphs.
Extensive experiments on long-range graphs understanding as well as long-range sequence mod- elling and point clouds with state-of-the-ar or competitive results along with ablation studies to demonstrate the effectiveness and efficacy of our method.
2 Related work
Our contribution is related to wide range of literature on deep learning on graphs, and in particular application of transformer architectures to graphs, as well as the line of research of efficient transformers. Bclow, we present a concise overview and further motivate our approach in the light of related literature:
Graph neural networks based on message- passigg scheme (Scarselli et al. 2009 Gileer et al. 2017 Kipf & Welling 2016 Xu et al., 2019 Corso et al.) 2020 Hamilton et al. 2017 Velickovid
Learning on graphs
2022) are currently the most widely adopted architectures for learning on graphs. By imposing a sparsity pattern derived from the input graph on the computation graph, this type of neural architecture benefits from linear complexity in the input size. While efficient and often competitive, the message-passingg scheme based on propagating and aggregating information among local neighbourhoods possesses well-known limitations such as too much focus locality (Morris et al. 2019 Xu et al. 2019) over-squashing (Alon
& &Yhavv 2021 Topping et al. 2022) and over-smoothing (Chen et al. 2020a Oono & Suzuki 2020). pre- venting them from dealing with long-range graphs (Dwivedi el. al. 2022b) or capluring multiscale structures
(Hy & Kondor 2023 Ying et al. 2018). In particular, several previous works on graph learning adaptively construct a hicrarchy using differentiable pooling (Gao & Jil 2019 Knyazev et al.] 2019 Khasahmadi et al., 2020 Lee et al., 2019b; Raajaan et al., 2019). Their main distraction with our work is that they construct soft cluster assignment, which does not help with computational complexity when working with trans- formers. Moreover, pooling operations only enable information sharing in bottom-up direction, while our hierarchical self-attention, enables information sharing between parents, children and siblings.
Recently, transformers built upon the self-attention mechanism, which takes into account all pairwise inter- actions between nodes, has been employed to alleviate these issues (Müller et al. 2023
Transformers for graphs Previous studics have adapted conventional transformers to work on graph domains by introducing novelties, such as graph feature encoding schemes Dwwivedi & Bresson 2020a
Rampasek et al. 2022: Ying et al.] (021), ) spectral attention (Kreuzer et al. 2021), ) graph tokenization (Kim
et al. /022a structural information extraction (Chen al.) 2022 Rong al. 2020), relative positional
encoding MMialon et al.) 2021). and attention on meta-paths (Yun al.) 2019 These techniques enable
transformers to display good performance on wide range of small graph benchmarks (Hu et al. 2020
Sterling & Irwin 2015). However, extensions of vanilla transformers, which retain quadratic complexity. encounter challenges in terms of memory and execution time when the graphs become larger (Dwivedi et al. 2022b). Our work aims at bridging the gap between graph transformers and message- passiing networks. Building graph hierarchies enables us to operate local attention at each level and achieve global attention by aggregating and updating information between the levels. This, as a result, helps with the communication between distant nodes, while benefiting from sub-quadratic complexity.
Efficient transformers for modeling long-range dependencies To case the quadratic complexity of vanilla transformer in modeling long sequences, several mechanisms were proposed to simplify the computa-
3

---

Published in Transactions on Machine Learning Research (04/2024)
tion of the attention matrix as summarized in survey (Lin et al. 2022]. Reformer (Kitaev et al. 2020a is content-based sparse attention transformer that matches tokens together into buckets using yocality-sessileee hashing before applying local attention. Others employ learnable sparsity patterns, like Sinkhorn transform- ers (Tay et al. 2020a) which induce sparsity by sorting the keys and values matrices such that local houristics the computation ion of the scaled-dot-produsee Most efficient transformers (Guo el al.) 2019)
can be applied
(Beltagy et al.) 2020b Ravula et al. 2020 Zaheer et al. (020b)) resort to similar sparsity heuristics to get rid of the bulk of the computations in the dot-product attention in structured way, allowing for efficient implementation without custom CUDA kernels. They often rely on local information processing, only se- lecting a few tokens as "global tokens" which will attend to every token and can be attended to by all other tokens. Further away from sparsity-based approaches, Transformer-Xi (Dai et al. 2019) is a memory-based transformer that adds recurrent connections across segments by reusing the hidden states of the model w while processing tokens of the previous segment. The work closest to ours is BP Transformer (Ye et al.) 2019); which uses binary partitioning of a sequence order to integrate information from long-term context in a fine-lo-carse fashion. Our model relies on sequential local peessage-psssingg updales in a tree, as opposed to BP Transformer, which achieves global information sharing by computing intermediate nodes' embeddings in parallel directly from their leaf nodes instead of their direct children. Also, it relies on the sequential ordering of the tokens to construct the dynamic hicrarchy, which is impossible to do on graph-structured data for which a natural monotonous ordering does not always exist.
We argue that models from the efficient transformers literature outlined above are not suitable for graph- structured data for at least three reasons:
Their approach to sparsification may rely on ID distance between tokens (e.g. convolutional-seuse locality) and thus cannot be applied to other input modalitics than soquences.
2. Their efficient implementation on GPUs often requires custom CUDA kernels, which limits the widespread availability of such models.
3. Most importantly, they often accomplish sparsification via mathematical tricks which may may not constitute natural or motivated inductive bias towards better generalization.
As a consequence, vanilla attention remains a strong statc-of-thhe-ar general-purpose model in terms of task performance for a wide array of input modalities. We believe that the main limitation towards better task performance of a general-purpose model is the ability to retrieve specific information from long-range interactions, which lost in the "noise" of the large sum over input tokens performed by attention pooling over large input data. In this paper, we aim not only to perform on par with vanilla attention on designated benchmarks but also to outperform classical attention in terms of task performance thanks to the powerful inductive bias provided by adaptive hicrarchical attention.
3 Method
Overall, our proposed methods comprise two interacting components: (i) learning hierarchical latent structure over the input tokens and (ii) applying a sparse attention scheme over the recursive k-ary tree computed in (i) in order to update tokens' embeddings.
The k-ary tree is built over the input tokens, which correspond to leaf nodes. The other intermediate levels of the tree up until the root node will henceforth be denoted virtual nodes. We use X E gaxd to denote the embedding of input tokens or node features, and E E R=xn for the weighted adjacency matrix - that is Ey > 0 implies an edge between node &s and uy with weight Egg. We use 7 = +(X,E) for a k-ary tree, where n = Σ yı is the total number of nodes in the tree, including both leaf nodes and internal or virtual nodes. Here, X E (O....,A 1} indexes the tree level, where = 0, X = A - indicates leaf level and root level, respoctively. Using this indexing, X2 E Raxxd xd the embeddings matrix of the nodes at level X. Similarly, Ex E R x corresponds to the adjacency matrix at that level. Finally, we use Ix C (O....,m} for node indices in given layer.
4

---

Published in Transactions on Machine Learning Research (04/2024)
Next, we elaborate on two components of our model, adaptive construction of hicrarchy and sparse self- attention guided by the resulting tree.
High-level nodes
X
Message Passing
Low-level nodes 15
GumbelMax
Cluster Index III
Low-level nodes X E
Figure 2: Our proposed Cluster Learning Block. In this block, the features of low-level nodes are used to learn the cluster presented as virtual high-level nodes.
3.1 Adaptive Hierarchy
Although Soquoia's k-ary tree be pre-constructed using partitioning algorithm designed especially for given input domain, provide approach to learn the clustering automatically. In contrast to previous works (Goo & Jil 2019 Knyazev et al. 2019 Khasahmadi et al., 2020 Lee et al.) [019b] Ranjan ct al. 2019) that learn a soft clustering assignment for nodes, ours learns to partition the inputs into mutually exclusive clusters and coarsen into the upper-level resolution in the hierarchy, which enables inter-level and intra-level attention at different resolutions.
A hard clustering of nodes into k clusters is a mapping π {I,..,, n) + {l,..., k} where T(1) = if node is assigned to cluster j. The clustering is represented by assignment matrix II f0, 10 where (g) Since require this hard assignment to be the output of neural network, we relax IT to P E 1)m*), row-stochastic matrix representing distribution over clusters. We then sample from this distribution to produce cluster assignment. To make this process differentiable, we employ the Gumbel- max trick (Gumbel 1954 Maddison et al. 2014 Jang et al. 2016) that provides a simple implementation and efficient way to define categorical distribution π as:
II onc-hot (argmax [9t, log Pt,e)),
where P. = .P...... P.,], is vector of class probabilities computed by Softmax, and g&d Gumbel 1) are i.dd samples from Gumbel distribution.
To construct the hierarchy in bottom-up fashion, starting from leaf nodes we use a Message-Passinng Neural (MPNN))(MPNNN the corresponding graph Ex),Ex) compute the probability matrix P and apply the GumbelMax function to derive II, and therefore the layer above in the hicrarchy:
Pa MPNN(XX,1,& a GumbelMax(P3).
As II, is differentiable, the parameters of MPNN can be updated via backpropagation, resulting in an effective end-to-end method for learning to cluster. For cach level X, corresponding node indices Ix can be easily retrieved from cluster memberships II, by converting from one-hot vector to index.
After identifying the cluster memberships in the layer above in the hierarchy, we the membership to aggregate the features of lower nodes and edges to construct higher-level node and edge features:
X3 II I Ex =T=
(1)
The interaction in the opposite direction is also sraighh-foowwade
Xx- =TXx-
(2)
5

---

Published in Transactions on Machine Learning Research (04/2024)
We term these two operations Pooling(Bo (Bq. and ExpandinggEq, 2 layers.
In short, when forming a tree, the nodes and edges at higher level are constructed based on the lower level. Specifically, after the input is fed into Adaptive Hierarchy, the block yields assignment matrix II. Based on this matrix and low-level nodes X and edges &, we can substitute them into Eq. to obtain the higher-level nodes and edges, resulting weighted graphs. Optionally, the equation can also applicable for edge attribute. However, in our implementation, for the simplicity, we keep the graph unweighted, which means only consider whether two nodes connected without considering the weight of the connection. The pseudocode for this process is presented in Alg.
3.2 Hierarchical
Block
Sequoia's hierarchical attention scheme attends three different neighbourhoods based on the tree structure:
Children Cx (all nodes j E [x-1 Parent[j]] i).
Siblings Sx (all nodes E Tx Pjrent[j]] = -aarent()). [o). Ancestors Ax, (all nodes j E Lx U. U...U E 1A-1
Parent o... Parent [ = j).
times
Our proposed attention scheme consists in applying:
ATTN(-) A ()) = ((44)))))))
where @ an appropriate fusion layer mixing the updated embeddings of each node according to the three different attention types ( (c- children, s siblings and a ancestors) on the node axis. Overall, for each node E Jx, we first gathers three types of neighbors j of target nodes i, including children CX.,, siblings Sx and ancestors Ax,; using the tree structure- illustrated in Figure 1 Then, compute three different types of embeddings A A², and A for the target node based on its atlention scores to each type of neighbor. These resulting embeddings are aggregated using pooling to produce a unifying embedding A for the target node.
More formally, let Qi be the query embodding associated with node i, and (K,, Vý)jev be the key and value embeddings associated with nodes j in the neighbourhood, then the attention for children AC,,, siblings A A ancestors A,,, of node i at level X in k-ary tree J is defined as:
(x3,0 = Attention Wg.Wg.We
Qii (Kj, Vill e caul,
( = Attention wg.wg.wy [Qi, (K,, Vi)s Sx
(x3, Attention Wy. Wy..g.Wp
[Qi (Kj, V/ys Ax.
We finally fuse the updated embeddings proposed for node by each of the three attention types using simple average pooling, which can also be or max pooling. Here, do not combine the embeddings of children, siblings and ancestors with the embedding of each node itself because they will be implicitly combined using residual connections mentioned in the next section.
A Puverage [A [A A A2,, A A AS + A + AL...
3
3.3 Putting it all together
Initially, the cluster learning block is applied repeatedly A 2 times to learn an A-layers tree. Here, the root node representation is given by pooling the XA-2, without any "Icarning". The term 'without any learning" means that the rool. node's embedding is not initialized as a learnable parameter that is updated via back- propagation, but is the aggregation of node embeddings the low-level nodes. After constructing the tree structures level by level, a combination of message passing, hicrarchical attention, and residual connections are used refine node features, before pooling (Eq. [ them to produce the features for the higher level; call these steps bottom-up blocks (Alg 2). By employing the bottom-up iteratively from the leaves up to

---

Published in Transactions on Machine Learning Research (04/2024)
Root node XT A-1 8
High-level nodes X3 Ex
Rottom-UIp
Top-Down
Low-level nodes X
Leaf nodes X; Ej
Figure 3: Our proposed Sequoia which mainly Bottom-Up and Top-Down block to aggregate local features from leaves to root and propagate global features in the opposite direction.
the root, the model can extract global information from local features. propagate back the information to local features, top-down block (Alg. 3) used. In this block the feature from the higher level needs be expanded to have the same shape the lower one. Subsequently, we apply MLP refined lower-level and then add both expanded (Eq. 2 and refined features together. The details of the combination of all mentioned blocks are presented in Alg. This block is also applied several times until the information reaches the leaf level. We refer the readers to the Appendix for time-complexity analysis.
Algorithm 1: Building Hierarchical Tree
Input: Xx-1,Ex-13
b Inputs are nodes and edges of low-level in the hierarchy
Output: Ex,Ex;
Outputs are nodes and edges of high-level in the hierarchy b Apply the MPNN to compute the probability matrix P IТ GumbelMax PA): a Apply the GumbelMax function to derive the assignment matrix II b Compute the high-level nodes Xx b Compute high-level edges Ex
PA = IPPNN(XA-LEA-1)
X, IIT
Ex 11T
Algorithm 2: Bottom-up Block
Input: XA-1, Ex-1;
Imput is nodes and edges of low-level in the hierarchy
Output: XA;
b Output is nodes of high-level in the hierarchy & Apply the MPNN to compute the local features Apply the ATTN to compute the global features Combine local and global features b Compute high-level nodes X,
Xlocul XA-1 + MPNN EA,11
X Xatobal XA-1 + ATTTN(X
X X1oeal + Xggoball
X, = POOLING(X,_))

---

b Aggregate features for nodes of higher level Published in Transactions on Machine Learning Research (04/2024)
Algorithm 3: Top-down Block
Input: Xx-1, Xx, Ex
Input is nodes and edges of high-level in the hierarchy
Output: X Xlocal = Xx + MPNN(XX):
> Output is nodes of low-level in the hierarchy
D Apply the MPNN to compute the local features Apply the ATTN to compute the global features > Combine local and global features b Compute new low-level nodes b Combine old and new low-level nodes to generate X
Xglobal = Xx + ATTN(XX)
X =Xlocal Xglobal
X1_ (NNOOOLIN(X)))
X A-1 = - -X
Algorithm 4: Entire Network
Input: Xo,80
Input is leaf nodes of the hierarchy b Output is nodes of all levels in the hierarchy
Output: X';
for (i = < X, 1) do
E:41 + Build Hierarchical Tree (X), Calculate nodes and edges of higher level
end
for
( = 0, < X, i= i do X1+1 + Bottom-up Block Ei)&;
end
for
(i= X, i> i= 1) do X Top-down Block (X,-1, X &
Aggregate features for nodes of lower level
end
4 Experiments
4.1 Graph understanding
Dataset Description To demonstrate the effectiveness of our hierarchical inductive bias, we apply Sequoia to three graph modeling datascts, namely LRGB (Dwivedi et al. 2022c) Polymer (St. John et al.. 2019 and Citation network (Sen et al. 2008):
LRGB [Dwivedi et al. 2022c is long-range graph benchmark that is built on peptides and images. In this work, we use 4 graph learning datasets: peptides-func (peptides), peptides-struce (pep- tides), VOC-sp (image), and COCO-sp (image). Table demonstrates the statistics of these datasets. In particular, the long-range property determined based on the graph diameter (i.e. the length of the shortest path between the most distant nodes).
Polymer (St. John et al. 2019) cointains 54,000 molecules associated to polymer properties. The task in this dataset is to predict three types of density functional theory (DFT) metrics (Hohenberg) & Kohn 1964) for each polymer. In particular, the model is trained to predict the first excitation energy of the monomer (GAP), the energy of the highest occupied molecular orbital (HOMO), and the lowest unoccupied molecular orbital (LUMO). The predictions are evaluated and compared with the two important benchmark error levels (Faber et al. 2017): (1) DFT errors, the estimated average nature; and (2) Chemical accuracy, a standard target error
error of the DFT approximation recognized within the chemistry community.
Citation networks (Sen et al. 2008) is benchmark comprising 3 subdatasets named Cora, Cite- seer, and Pubmed. In cach dataset, documents are represented by sparse bag-of-words fcature vectors which correspond to the nodes of the underlying graph. Besides, the edges of the graph are the citation links between documents.
The same configuration shared across all datascts. In particular, we set batch size 128, hidden dimension d 96, tree's layer A 3, stochastic gradient descent optimizer with the initial learning rate of 10-1, maximum number of clusters C 32, and epoch 200.
Implementation Details
8

---

Published in Transactions on Machine Learning Research (04/2024)
Table Description of datasets in the LRGB benchmark
Dataset
# Graphs
# Nodes
# Edge
Diameter
Peptides-func
15k
2,3m
4.7m
56.99 + 28.72
Peptides-strue
15k
2,3m
4.7m
56.99 28.72
VOC-SP
11k
5m
30.7m
27.62+2.13
COCO-SP
123k
58m
332m
27.3911.11
Table 2: Results peptides property prediction. The baseline results are taken from Dwivedi et al.
2022b
Peptides-struct
Peptides-func
Method
Params
Time
MAE
AP 1
GCN (iip & Welling 2016
508k
3s
0.3496
0.0013
0.5930
0.0023
GCNI CChen et al. 2020b
505k
3s
0.3471 0.0010
0.5543
0.0078
GINE (Xu et al.
2019)
476k
3s
0.3547 0.0045
0.5498
0.0079
GatedGCN Bresson & Laurent 2017
509k
4.5s
0.3420
0.0013
0.5864 + 0.0077
GatedGCN RWPE Bresson & Laurent
2017
506k
4.5s
0.3357
0.0006
0.6069 = 0.0035
Transformer LapPE Vaswani et al.
2017
488k
7.5s
0.2529
0.0016
0.6326 0.0126
Transformer + RWPE Vaswani et al.
2017
488k
7.5s
0.2620 + 0.0010
0.6502 0.0101
SAN LapPE Kreuzer et al.
2021
493k
55s
0.2683 0.0043
0.6384 0.0121
SAN + RWPE Kreuzer et
2021
500k
55s
0.2545 + 0.0012
0.6562 0.0075
GPS Rampášek et al. 2022
500k
14s
0.2500
0.0005
0.6535 = 0.0041
MLP-Mixer (He et al.
2023
396k
7.0s
0.2478
0.0010
0.6970 0.0080
Sequoia + RWPE (ours)
346k
5.3s
0.2453
0.0006
0.6755 0.0074
Sequoia LapPE (ours)
346k
5.3s
0.2526 + 0.0019
0.6323 + 0.0042
Table 3: Results VOC and COCO datasets. The baseline results are taken from
Dwivedi et. al.
2022b
VOC
COCO
Method
Params
Time
macro F1 f
Params
Time
macro F1 f
GCN
Kipf Welling
2016
496k
10.6s
0.1268
0.0060
509k
116.3s
0.0841
0.0010
GCNI
Chen et al.
20206
192k
8 8.8s
0,1698
0.0080
505k
111.2s
0.1404
0.0011
GINE
Nu al.
2019
505k
8.9s
0.1265
0.0076
515k
105s
0.1339
0.0044
GatedGUN
Bresson
Laurent.
502k
18.5e
0.1265
509k
189.7
0.2641
GatedGCN
IWTE Bressnn
Laurent 2017
502k
18.5s
0.1265
0.0076
509k
189.7s
0.2574
0.0034
Transformer LapPE Vaswani
501k
U.2694
SU8k
196.1s
0.2618
Transformer RWPE (Vaswanl et
2017
501k
16.1s
0,2718
0.0076
508k
196.15
0,2686
0.0027
SAN + LapPE Kreuzer et sl.
2021
531k
185s
0.3230 + 0.0039
536k
2148s
0,2592 + 0.0158*
SAN RWPE (Kreuzer
2021
468k
185s
0.3216
0.0027
474k
2148s
0.2434 + 0.0156*
GPS Rampádek et sl.
510k
23.2s
0.3748
0.0109
510k
280.5s
0.3412
0.0044
Sequoia RWPE (ours)
438k
13.5s
0.3379
438k
157.8s
0.2726
Sequoia
LapPE (ours)
438k
13.5s
0.3481
0.0116
438k
137.8s
0.2756
0.0027
In graph-level classification, there is one layer of message passing to refine the feature before constructing the hierarchy, while in node-level segmentation, we apply four message-passing layers in this process. Thereafter we employ a combination of message-passing hierarchical attention and residual layers once at each level of the tree to aggregate the global and local information. Additionally, we project the positional encoding (c.g., Laplacian (Dwivedi & Bresson 2020b) or random walk (Dwivedi al. 2022a)) to the same dimension of features (d 96) and add them into the features.
Results Our model consistently outperforms other models in the benchmarks presented above. Regarding effective time complexity, our model is slower than message passing, yet faster than other transformer models

---

Published in Transactions on Machine Learning Research (04/2024)
for graphs. More precisely, in the peptides dataset Table 2 successfully reduce the MAE below 0.25 and surpass the other methods by a large margin, suggesting that our Cluster Learning Block constructs a useful hierarchy. We observe the same performance for the polymer dataset Table 4 We believe that while a learnable hierarchy breaks the graph into simpler and more meaningful subgraphs, the aggregation function, with Bottom-Up and Top-Down techniques, enables efficient communication among features from different levels in the hicrarchical structure. This reasoning may hold true for the polymer dataset because this data is related to chemistry, where learning from subgraphs highly benefits the network. For node-level classification, as shown in Table Sequoia achieves better performances in VOC and COCO. However, the improvements are marginal for citation networks Table G This aligns with the fact (Chen et al.) 2020b; modeling methods on these benchmarks.
Dwivedi et al., 2022b) that shallow GNNs are more effective than their deep counterparts and long-range
Table 4: Results on polymer property prediction. We trained all the following baselines for the Polymer datasets.
GAP
HOMO
LUMO
Method
Params 4
Time
MAE +
MAE 4
MAE
DFT errur
1.2
2.0
2.6
Chemical accuracy
0.043
0.043
0.043
GCN
Kipf & Welling
2016
527k
13s
0.1094 + 0.0020
0.0648 = 0.0005
0.0864 = 0.0014
GCN
Virtual Node
Kipl & Welling
2016
557k
13s
0,0589 + 0.0001
0.0458 = 0.0007
0.0482
0.0010
GINE
et al.
2019
527k
0.1018 + 0.0026
0.0749 + 0.0042
0.0764
0.0028
GINE
Virtual Node
Xu et al.
2019
557k
11s
0.0870 0.0040
0.0565 0.0050
0.0524 0.0010
Transformer + LapPE Vaswani et sl.
2017
320k
21s
0.0541 = 0.0029
0.0445 = 0.0020
0.0458 0.0057
Transformer + RWPE Vaswant et a.
320k
0.1280 0.0122
0.0854 0.0053
0.0990 0.0054
SAN LapPE Krewzer el
2021
495k
151s
0.0461 + 0.0022
0.0356
0.0026
0.0356 = 0.0017
SAN RWPE
Kreuser et al
2121
495k
151s
0.0467 + 0.0031
0.0351 = 0.0024
0.0372 = 0.0011
Sequoia LapPE (ours)
346k
19s
0.0494 + 0.0010
0.0384 0.0028
0.0376
0.0035
Sequoia + RWPE (ours)
346k
19s
0.0409 + 0.0009
0.0312 = 0.0014
0.0312 = 0.0012
Table 5: Sequence Classification results on Long Range Arena benchmark for efficient transformers.
IMDB
Cifar10
Listops
Retrieval
Patshinder
Method
Accuracy
Accuracy
Accuracy 1
Accuracy
Accuracy
Complexity +
BigBird Zaheer et al.
2020a
64.02
40.83
36.05
59.29
74.87
O(n)
LongFormer (Beltagy ct al. 2020a
62.85
42.22
35.63
56.89
69.71
O(n)
Performer (Choromanki et al
2021
65.40
42.77
18.01
53.82
77.05
O(n)
LinFormer [ang al.
2020
53.94
38.56
35.70
52.27
76.34
O(n)
Local Attention
Tay
20c
52.98
41.46
15.82
53.59
66.63
O(n)
Linear Trans
Tay et al.
2020c
65.90
42.34
16.13
53.09
75.30
O(n)
S4
(Gu cl. al.
2022
76.02
86.09
88.65
87.09
86.05
(n
Mega Ma al.
2023
90.43
90.44
63.14
91.25
96.01
O(n)
Soquoia (ours)
75.10
49.88
37.70
67.00
87.30
O On n)
4.2
Long Sequence Modeling
Dataset Description We evaluate Sequoia on Long Range Arena (Ty et al. 2020b (sequence classifi- cation on multiple input modalities) to demonstrate the effectiveness of our model. Our goal in the sequence modeling experiments is to validate that Sequoia is general-purpose model and can handle multiple input modalities, including sequences of discrete Lokens. Since our objective is to propose a hierarchical method that can scale to large-sized data (e.g., long sequences, large graphs, etc.) efficiently, we compare Sequoia with several efficient Transformers, including BigBird (Zahcer et al.) 2020a) LongFormer (Beltagy et al. 2020a), Performer (Choromanski et al.) 2021), LinFormer (Wang et al. 2020 Local Attention (Tay et al. 2020c3, , and Linear Transformer (Tay et al. 2020c).
10

---

Published in Transactions on Machine Learning Research (04/2024)
Table Results on Cora, Citeseer, Pubmed
Citeseer
Cora
Pubmed
Method
Acc
Acc
Acc f
Runtime
TSVM (Joachims 1999
0.640
0.575
0.622
[P(Zhu et al. 2003)
0.453
0.680
0.630
GRÄPHEME (Tian et al. 2014
0.432
0.672
0.653
PLANETOID-G Yang
2016
0.493
0.691
0.664
PLANETOID-T
Yang et al.
2016
0.629
0.757
0.757
ChebNet DDfforrard et al. 2016
0.698
0.780
0.744
GCN Kipl Welling
2016
0.703
0.815
0.790
0.32s
GAT Veličković et al.
2018
0.725
0.830
0.790
2.42s
Sequoia (ours)
0.732
0.845
0.794
2.15s
Implementation Details For experimental simplicity, we use a standard model backbone configuration throughout all sequence classification lasks. More specifically, our model consists of four transformer layers, with a hidden dimension of 512 splits across settings for all datasets. We use dropout rate of 0.1, and our model optimized by Adam optimizer with a learning rate of 0.001 with linear warmup. The accuracy reported corresponds to the test accuracy obtained on the best-performing model according to validation steps over every 2,000 training steps.
Results Table reports our results for long-range sequence modelling and by the introduction of hierar- chical interactions, Sequoia surpasses other conventional efficient transformers on LRA. One possible reason for the improvements that during forming hierarchy, we explicitly divide sentences or images into smaller chunks which facilitates aggregation process of the model. Otherwise, other approaches in the table let the network process entire sentences or images.
4.3 Point clouds
Dataset Description In order to demonstrate the elficiency of Sequoia on point clouds, we conduct experiments on two tasks:
Shape classification: ModelNet40 (Wu et al. 2015 dataset is classification dataset that contains 12,311 3D models categorised into 40 classes.
Part segmentation: ShapeNetParl (Mo el al.
2019 dataset is a part segmentation dataset, which
contains 16,881 synthetic point clouds from 16 classes.
Implementation Details In all experiments, our model use four Transformer layers and 4-layer tree. The other configurations are kept the same as other works. In particular, the optimizer used in both experiments is SGD (learning rate = 0.05, momentum = 0.9, and weight decay 0.0001) and we train the model for 200 epochs with batch size of for classification and 16 for segmentation.
Results As observed from Table 7 and g Sequoia achieves competitive results on point cloud classification, but it does not seem to perform as well as other baselines in part segmenlation. The reason is that apart from PointNet (Qi et al. 20177),, the remaining models in this experiment already use hierarchical structure in their design, so our method obtains no significant superior in terms of aggregating features, compared them. However, with the new algorithm for tree construction, we can avoid O(n?) complexity when caleulating distance for sampling and finding nearest neighbors in other point-based methods.
11

---

Published in Transactions on Machine Learning Research (04/2024)
Table 7: Shape Classification results on ModelNet40
Method
Accuracy
mAcc
Params 4
Time
VoxNet
Maturana & Scherer
2015
85.9
83.0
MVCNN
[Su et al.
2015
90.1
PointNet
Qi et al.
2017a
89.2
86.0
0.6M
0.8lms
DGCNN
Wang et
2019
92.9
1.82M
GridGCN Xu et al.
2020
93.1
91.3
42.2ms
Set Transformer Leee et a
2019a
90.4
PointNet Qii et al 2017b
91.9
88.4
1.0M
1.98ms
PointConv (Wu al.
2019
92.5
KPConv [ Thomas al. 2019
92.9
14.3M
7ms
Trans'orra
ZZhao et al.
2021
93.7
90.6
11.7M
2.0ms
Sequoia (ours)
92.0
88.4
10.9M
9.6ms
Table 8: Part Segmentation results on ShapeNetPart
Method
Inst IoU t
Class IoU 1
Params
Time
PointNet
Qi et
2017a
71.9
43.7
3.6M
1.0
DGCNN
Wang et al.
2019)
85.1
82.3
1.3M
125ms
PointNet
[Qi al.
2017b
85.1
81.9
1.0M
1.41ms
PointConv
Wu et al.
2019
85.7
82.6
PointCNN
(Li et al.
2018)
86.1
84.6
Point Tranntoreeerrrrr
Zhao et al.
2021
86.6
83.7
7.8M
2.69ms
Sequoia (ours)
83.8
80.6
7.26M
7.8ms
4.4 Ablation study
To comprehensively evaluate our model, we conduct ablation studies on several aspects including graph sizes, the maximum number of clusters, different types of hicrarchy, the number of layers, and diffcrent types of allention blocks.
Table 9: Ablation study for complexity and memory.
Synthetic Graph
Method
Nodes
Time
Memory
Parameters
Message Passing
0.0156s
1.171GB
350k
Attention
100
0.0234s
1,17GBB
500k
Sequoia (ours)
0.0221s
1.187GB
437k
Message Passing
0.0237s
L447GBB
350k
Attention
1000
0.0379s
1,635GB
500k
Sequoia (ours)
0.0235s
1.355GB
437k
Message Passing
0.5152
20.295GB
350k
Attention
10000
00M
0OM
500k
Sequoia (ours)
0.5081
19.563GB
437k
Scalability
We generate synthetic graphs with varying sizes from 100 to 10,000 nodes and perform node
classification. Table shoows that our model is comparable to messago-passing networks in terms of efficiency.
Different types of hierarchy
We compare 3 types of hierarchy in Table T0
12

---

Published in Transactions on Machine Learning Research (04/2024)
Table 10:
Ablation study on clusters and hierarchy.
Peptides-fune
VOC
Peptides-func
VOC
Cluster
Hierarchical
Accuracy
macro F1
Accuracy
macro F1 f
16
66.95
0.3236
Fixed
62.78
31.83
67.55
0.3481 0.3435
Random
66.19
31.20
64
65.67
Learnable
67.55
34.81
Table 11:
Ablation study on trec's layers and attention blocks.
ModelNet40
ModelNet:10
# Layers
Attention Blocks
Accuracy t
Accuracy t
3
90.19
Children
87.50
4
92.00
Children + Siblings
91.50
5
89.87
Children Siblings Ancestors
92.00
Table 12:
Ablation study hierarchical structure
Method
Inst IoU
Class IoU
Non-hierarchy
78.8
74.9
Sequoia's 's hierarchy
82.7
79.5
In a fixed hierarchical structure, we initially assign nodes to a predetermined hierarchical frame- work, where clusters are defined based on the type of the dataset (e.g. KNN for point clouds, k ring for graphs, sentence chunking for texts, and sliding window for images).
In a random hierarchical structure, we assign nodes a randomly generated hierarchical frame- work.
In our learnable hierarchical structure, the model learns the optimized hierarchical structure.
By constructing the meaningful hicrarchical structure, our methods achieves best performance, compared to the others.
Other configurations We run our model on several settings in terms of the maximum number of clusters, the number of layers, and different lypes of attention blocks. The results are reported in Table TO and
4.5 Other implementation details
Fairness comparison between our method and the baselines For Tables 3 we reran all the models in the benchmark using the same framework GraphGPS (Rampášek et al. 2022 on same devices (RTX3090) for fair comparison terms of runtime and number of parameters. 2019) to evaluate the run-time for Table 6 There is no unified framework for point clouds, so we mainly obtained information based the benchmarks of other papers such PointNext
use (Fey Lenssen
al.) 2022) and RepSurf(Ran el al.) 2022) for fair comparison in Table 7 and g As for Table 5) since our code does not use the same framework as other efficient transformers, we to report the Big-O complexity instead. In terms of the training time, we followed the same configuration when training and all our experiments had the same number of epochs as the basclines, but the time of each epoch was different.
Tuning the maximum number of clusters In the Adaptive Hicrarchy, the number of cluster is predefined meaningfully based on the statistics computed from the dataset. For example, & may be the number of functional groups existing in molecule, the number of parts of 3D objects, the number of
13

---

Published in Transactions on Machine Learning Research (04/2024)
elements inside sentence. Furthermore, it is unnecessary to define & precisely since Adaptive Hierarchy block allows for empty clusters, which means we only need to approxinate the upper bound of the number of actual clusters. Additionally, the challenge of defining k could be eased by the multi-level mechanism of the structure. Specifically, when the number of the predefined clusters is too small to capture the entire functional sub-graphs, they can be extracted at higher level in the tree. In short, during the experiments, we merely select a reasonable upper-bound & that fits does not cause GPU memory issues rather than putting effort into failoring the best structure by tuning for each level in the hierarchy.
The flexibility of Sequoia's Adaptive Hierarchy Indeed, our Adaptive Hierarchy a standalone block that can be potentially viewed plug-and-play module for other graph transformer models. Theoretically, the hierarchical structure can potentially benefit other methods that require hierarchical organization of the data, In our paper, our primary proposal is hierarchical "attention", aimed at maximizing interactions between different levels in the hicrarchy generated by the Adaptive Hicrarchy block, which can be casily integrated to other architectures.
Conclusion
The first ingredient of our model, self-attention, has proved expressive and invaluable in learning powerful models on a varicty structured data. The second ingredient, hicrarchy, provides a useful inductive bias and provides a convenient structure for both the organization of representations and the optimization of computational budget. This paper introduces mechanism for merging these in a single geometric learning building block. We found this combination interactions. Our ablation studies suggest that in terms of computation, it compares with pessagg-passing methods, even surpassing them larger graphs.
be particularly useful for graphs that involve long-range
Acknowledgements
SR and HS want to acknowledge funding support from FPT, CIFAR, and NSERC. Computational resources were in part provided by Mila and Compute Canada.
References
Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. Tn International Conference on Learning Representations, 2021. URL http:p////perreeeee..eeeefreeeee net./foummid==
1800PhOCVH2
Iz Beltagy, Matthow E. Peters, and Arman Cohan. arXiv:2004.05133 2020a.
Longformer:
The long-document transformer.
Ix Beltagy, Malthew E Peters, and Arman Cohan. Longformer: The long-document transformer. preprint axX1e:2004.05150, 2020b.
arXio
Xavier Bresson and Thomas Laurent. Residual gated graph convnets. CoRR, abs/1711.07553, 2017. URL http://arxiv org/
07553
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):3438 3445, Apr. 2020a. doi: ..1609/aaai.v34ii0 aaai. maai.org/ineg porg/index... waceeexx.ppp/AAAe
URL https://ojs.
Dexiong Chen, Leslie O'Bray, and Karsten M. Borgwardt. Structure-aware transformer for graph represen- tation learning. In International Conference on Machine Learning, 2022.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference
14

---

Published in Transactions on Machine Learning Research (04/2024)
on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1725-1735. PMLR, 13-18 Jul 2020b. URL htttps:/proceeddingg www.mmlrr.p... wttps/ppssss/////eeeeee/e htmml
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quiney Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Represenlations, 2021. URL. http:///pppppeeeeeeeeeeeerrrrrrr
Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pictro Lió, and Petar Velickovie. Principal neighbourhood aggregation for graph nets. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.00 028600 2019.
Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Re. Flashattention: Fast and memory-effiiient exact attention with 10-awareness. In Alice H. Oh, Alekh Agarwal, Danielle Bel- grave, and Kyunghyun Cho (ods.), Advances in Neural Information Processing Systems, 2022. URL
https://openreview..n htt.://wwww.......oooooooodtdaoa
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Proceedings of the 30th International Conference on Neural Infor- mation Processing Systems, NIPS 16, pp. 3844 3852, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. ArXiv, abs/2012.09699, 2020a.
Vijay Prakash Dwivedi and Xavior Bresson. A gencralization of transformer networks to graphs. CoRR, abs/2012.09699, 2020b. URL httts:://arii:i
htt.://gg.///rr..vorr///////0000 wttp:/a////...000033333 09699
Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshma Bengio, and Xavier Bresson. Graph neural networks with learnable structural and positional representations. In International Conference on Learning Representations, 2022a. URL htttps///ppaaaoooreeeeee////dddd
Vijay Prakash Dwivedi, Ladislav Rampášek, Michael Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. Long range graph benchmark. In S. Koyejo, S. Mohamed, A. Agarwal, D). Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 22326 22340. Curran Associates, Inc., 2022b. URL https::/proceedding www.eeeddninnnnnnnnsspeeee cetuurps.cc/papeeeeee wttpcccppppaeeeeeeee ficeerrfllssseeeeeeeeeeeeeeeeeer files/paper/ (000://H222226600000066777666662
and Benchmarks .pdf.
Vijay Prakash Dwivedi, Ladislav Rampášek, Mikhail Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. Long range graph benchmark. arXtv:2206.08164, 2022c.
Felix A Faber, Luke Hutchison, Bing Huang, Justin Gilmer, Samuel S Schoenholz, George Dahl, Oriol Vinyals, Steven Kearnes, Patrick F Riley, and O Anatole von Lilienfeld. Machine learning prediction errors better than dft accuracy. arXiv preprint ar arXiv: 1022. 05532, 2017.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR 2019 Workshop on Representation Learning on Graphs and Manifolds, 2019. URL https://arxiv.c orgg/ abs/ 0ttp/9933.02427
Hongyang Gao and Shuiwang Ji. Graph u-nets. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Ma- chine Learning Research, pp. 2083-2092. PMLR, 09-15 Jun 2019. URL hntps://proceediings..
preas/v97/gao13 /ww.99///a99..... html
15

---

Published in Transactions on Machine Learning Research (04/2024)
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning Volume 70, 1CML'17, pp. 1263 1272. JMgR.org, 2017.
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022. URL http:://oppnrrriee..nss
forum?id=uYLLozile
Emil Julius Gumbel. Statistical theory of extreme values and some practical applications: series of lectures, volume 33. US Government, Printing Office, 1954.
Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R. Martin, and Shi-Min Hu. Pet: Point cloud transformer. Computational Visual Media, 7(2):187-199, Apr 2021. ISSN 2096-0662. doi: 10.1007/s41095--22--000
URL http://dx. doi
Qipeng Guo, Xipeng Qiu, Penglei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. Star-transformer. arXiv preprint 19t::1990.0111 2019.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In I. Guyon, U. Von Luxburg, S. Bengio, Н. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: attps://procees of BucBbbaBSaadddlbbaaa//rrraaaaaaa
neurips wttps/errcppppppppeeeeeeee Files//pppee/21///1l/
Xiaoxin He, Bryan Hooi, Thomas Laurent, Adam Perold, Yann Lecun, and Xavier Bresson. A generalization to graphs. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 12724-12745. PMLR, 23-29 Jul 2023. URL https::/procceddiggg mlr. ppress/v22/hhe/// html
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention multidimensional transformers, 2020. URL httppp//pppaaeeeeeeeereerrrrrrrp
P. Hohenberg and W. Kohn. Inhomogeneons electron gas. Phys. Rev., 136:B864-BB8 B871, Nov 1964. doi: 136. B864. URL https://link: aps wrwp//o////ooo.o///o//ooo/ooooot http:///o.//...111..a.aaee.... B864
Weihua Hu, Matthias Fey, Marinka Zitník, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 22118 22133. Curran Associates, Inc., 2020. URI http:ts:// neurips. cc/ cc/paper files/
Truong Son Hy and Risi Kondor. Multiresolution equivariant graph variational autoencoder. Machine Learning: Science and Technology, 4(1):01503,, mar 2023. doi: 0001188//6322213///aa URL https: /ttpi/.r////.../////52//////
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:161100114, 2016.
Thorsten Joachims. Transductive inference for text classification using support vector machines. In Pro- ceedings of the Siateenth International Conference on Machine Learning, ICML '99, pp. 200 209, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc. ISBN 1558606122.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learn- ing Research, pp. 5166-5165. PMLR, 13-18 Jul 2020. URL katharropoulos20..htm
htt.://ociiinna.....rrre///////. press/v119/
16

---

Published in Transactions on Machine Learning Research (04/2024)
Amir Hoscin Khasahmadi, Kaveh Hassani, Parsa Moradi, Leo Lee, and Quaid Morris. Memory-based graph networks. In International Conference on Learning Representations, 2020. URL. htttps://peenrevier
het/forum?idd=1laleeerr
Jinwoo Kim, Dat Nguyen, Sconwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, and Seunghoon Hong. Pure transformers are powerful graph learners. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advonces in Newral Informalion Processing Systems, volume 35, pp. 14582- 14595. Curran Associates, Inc., 2022a. URL httpps://prooceediggs. httpe:///poceainngss.uurrrrspppp weteurppssccppppeefeeeeeee
pdf
Jinwoo Kim, Dat Tien Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, and Seunghoon Hong. Pure transformers are powerful graph learners. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Newral Information Processing Systems, 2022b. URL https: :
Thomas N Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. arXiv preprint 160: 02907, 2016.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International
Conference Learning Representations, 2020a. URL
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International
Conference on Learning Representations, 2020b. URL
Boris Knyazev, Graham W Taylor, and Mohamed Amer. Understanding attention and generaliza- tion in graph neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'AlchéBBue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL https://proceeddngg neurips. wwwppppprrcppppp.r.ppeppp httpr/pps.c/pppppe.l.l/////pppps /tles//pppee/20///ttlll Paper.
Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rothinking graph transformers with spectral attention. In M. Ranzalo, A. Beygelzimer, Y. Dauphin, P.S. Liang, and
J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, volume 34, pp. 21618- httpc//aaper..ilssppapperr//
21629. Curran Associates, Inc., 2021. URL
pdf
Xin Lai, Jianhui Liu, Li Jiang, Liwei Wang, Hengshuang Zhao, Shu Liu, Xiaojuan Qi, and Jiaya Jia. Stratified transformer for 3d point cloud segmentation. In Proceedings of the IEEE/CT DEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1000-8509, 2022.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutationninnarte machine learning, pp. 3744 3753.. PMLR, 2019a.
neural networks. In International conference on
Junhyun Lee, Inyeop Lee, and Jaewoo Kang. Self-attention graph pooling. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, vol- ume 97 of Proceedings of Machine Learning Research, pp. 3734-3743. PMLR, 09-15 Jun 2019b. URL
https::/proceediigg mir. wtwoss/v97/1ee111 html
Yangyan Li, Rui Bu, Mingehao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolution on x-transformed points. Advances in neural information processing systems, 31, 2018.
Tianyang Lin, Yuxin Wang, Xiangyang Tiu, and Xipeng Qiu. A survey of transformers. AT Open, 3: 111 132, 2022. ISSN 2666-6510. doi: http:://doo.orrg/00.00////// sciencedirect. httpa//teccccccccecc....c.eeee/n
URL https://www.
Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point- Vozel CNN for Efficient 3D Deep Learning. Curran Associates Tnc., Red Hook, NY, USA, 2019.
17

---

Published in Transactions on Machine Learning Research (04/2024)
Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention. In The Eleventh International Conference on Learning Representations, 2023. URL https://opeereviee net//orrum?idgglle3ccc
Chris J Maddison, Daniel Tarlow, and Tom Minka. A sampling. Advances in neural information processing systees, 27, 2014.
Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real-time object rocognition. In 2015 JEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 922 22-9928.7
TEEE, 2015.
Grégoire Mialon, Dexiong Chen, Margot Selosse, and Julien Mairal. Graphit: Encoding graph struct ure in transformers, 2021.
Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hicrarchical part-level 3d object understanding. In Proceedings of the IEEE/CVE conference on computer vision and paltern recognition, pp. 909 918, 2019.
Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the Thirly-Thann Third AA Conference on Artificial Intelligence and Thirty- First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, 1LAA11/IAAA119// v33i01 33013300222
AAAI Press, 2019. ISBN 978-1-57735-809-1. doi: 10.1609 /aaii. URL http:::/ddoi...rr... 1wwpg//....100///// 1609//6a0..... v33101 33014602
Luis Müller, Christopher Morris, Mikhail Galkin, and Ladislav Rampášek. Atlending to Graph Transformers. Arziv preprint, 2023.
Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classifi- cation. In International Conference on Learning Represendations, 2020. URL httts::/oppenrevle:: net/
forum?id=S11
Chunghyun Park, Yoonwoo Jcong, Minsu Cho, and Jacsik Park. Fast Point Transformer. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++:: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017b.
Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnct++ with improved training and scaling strategics. In Advances in Neural Information Processing Systems (NewrIPS), 2022.
Ladislav Rampasek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for general, powerful, scalable graph transformer. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (cds.), Advances in Neural Information Processing Systems, 2022.
URL https:://openrevier wwwaeeeeeeeeeeooooo/iaaaaaaaaaaa
Ladislav Rampášck, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for General, Powerful, Scalable Graph Transformer. Advances in Neural Information Processing Systems, 35, 2022.
H. Ran, J. Liu, and C. Wang. Surface representation for point clouds. In 2022 IEEE/CVF Con- ference on Computer Vision and Pattern Recognation (CVPR), pp. 18920-18930, Los Alamitos, CA, USA, jun 2022. IEEE Computer Society. doi: 10.1109//CV /ww.1100///V222888.... 222222888.0222118877. URL https:///oi. htt.ccppptttroccctet............
01837
18

---

Published in Transactions on Machine Learning Research (04/2024)
Ekagra Ranjan, Soumya Sanyal, and Partha Pratim Talukdar. Asap: Adaptive structure aware pooling for learning hierarchical graph representations. In AAAT Conference on Artificial Intelligence, 2019.
Anirudh Ravula, Chris Alberti, Joshua Ainslie, Li Yang, Philip Minh Pham, Qifan Wang, Santiago Ontanon, Sumit Kumar Sanghai, Vaclav Cvicek, and Zach Fisher. Etc: Encoding long and structured inputs in transformers. In 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020), 2020. URL htttlh:tp///// htttps:/www.allweb.oorm www.orrrgnnoooo/r//////o//o
emnlp-maine 19.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang, Self- supervised graph transformer on large-scale molecular data. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Compulational Linguistics, 9:53 68, 2021.
doi: 10.1162 /tacllcll 00353. URL
wwwp://ogg.rrg/22...... taco-.1
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neurol Networks, 20(1):61-88) 80, 2009. doi: 10.1109/TTNN 2008. 2005600.
Prithviraj Sen, Galileo Namata, Mustafa Bilgie, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Col- lective classification in network data. Al Magazine, 29(3):93, Sep. 2008. doi: 10. aamaaaaaaaaaa... v2913.2157.
URL https://ojs.aam aaai. orr/ httg///maazzinee/// index. php/
Peter C St. John, Caleb Phillips, Travis W Kemper, A Nolan Wilson, Yanfei Guan, Michael Crowley, Mark R Nimlos, and Ross Larsen. Message-passing neural networks for high-throughput polymer screening. The Journal of chemical physics, 150(23):234111 2019.
Teague Sterling and John Irwin. Zine 15- ligand discovery for everyone. Journal of chemical information and modeling, 55, 10 2015. doi: 10.1021 /a021///s..iiiit hcs.jeimm.bb0555..
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Mille Multi-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision, pp. 945-953, 2015.
Yi Tay, Dara Bahri, Lin Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. In Interna- lional Conference on Machine Learning, pp. 9438 9447. PMLR, 2020a.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint XrXiv:201..0 1.04006, 2020b.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv 2020c.
preprint XrXie:2011.0/4
Hugues Thomas, Charles R Qi, -aamanell Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 6411 6420, 2019.
Fei Tian, Bin Gao, Qing Cui, Enhong Chen, and Tie Liu. Learning deep representations for graph clustering. Proceedings of the AAAI Conference on Artificial Intelligence, 28(1), Jun. 2014. doi: 10.1609/ v281..2811.89) URL https://ojs http::///jss.aaa..... aaai. aaaai.rrr/indee http://e..pp//./aaaaaaaae///////
Jake Topping, Francesco Di Giovanni, Benjamin Paul Chanberlain, Xiaowen Dong, and Michael M. Bron- stein. Understanding squashing and bottlenocks on graphs via curvature. In International Conference on Learning Representations, 2022. URL. httpp:://openeevvie..eeef/ooooo
19

---

Published in Transactions on Machine Learning Research (04/2024)
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579 2500) 2008. URL http:://ww.aooooeppppsess///saaa html
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Petar Veličković. Message passing all the way up. arviv preprint or arXi::2202.110 11097,, 2022.
Pelar Veličkovié, Guillem Cucurull, Aranixa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018. URL https:
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. ar preprint arXiv:2006. 047,8, 2020.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph can for learning on point clouds. Acm Transactions On Graphics (tog), 38(5):1 12, 2019.
Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 9621-9630, 2019.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: deep representation for volumetric shapes. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912 1920, 2015.
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glemn Fung, Yin Li, and Vikas Singh. Nyströmformer: nyström- based algorithm for approximating self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 14138 14148, 2021.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL attps://openrevieee http://nwre.tttteeeoooonttttttaa
Qiangeng Xu, Xudong Sun, Cho-Ying Wu, Panqu Wang, and Ulrich Neumann. Grid-gen for fast and scalable point cloud learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 5661 5670, 2020.
Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML'16, pp. 40 48. JM,R.org, 2016.
Zihao Ye, Qipeng Guo, Quan Gan, Xipeng Qiu, and Zheng Zhang. Bp-transformer: Modelling long-range context via binary partitioning, 2019. URL https: /ttp///axvvvvv./////a wtgg/ba////aee htt.s///91.....
Chengxuan Ying, Tianle Cai, Shengjic Luo, Shuxin Zheng, Guolin Ke, Di Hc, Yanming Shen, and Tic-Yan Liu. Do transformers really perform badly for graph representation? In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, 2021. URL
Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchi- cal graph representation learning with differentiable pooling. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (ods.), Advances in Neural Information Processing Sys- tems, volume 31. Curran Associates, Inc., 2018. URL /ttp:sUURRtttpp:p/oo///eeiieppp www.oeeddiinnnn.uisss.... retpaas..arrrrrssss.cpppp.err oppp://0001177//attp22ccccceeeee
20

---

Published in Transactions on Machine Learning Research (04/2024)
Seongjun Yun, Minbyul Jeong, Rachyun Kim, Jaewoo Kang, and Hyunwoo J Kim. Graph trans- former networks. In H. Wallach, H. Larochelle, A. Beygelzimer F. d' d'Alch-Buuc, E. Fox, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL Be6300abb7777700044455555b---rrr
neurips seurips.cc/paper ftles//ppprr2019//11ll pdf.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Amirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Trans- formers for longer sequences. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 17283 17297. Curran Associates, Inc., 2020a. URL https://proceeddngss c6512112222244995351aaaaaaaaaa aa..a999-/7.a111aapapaee. ieerrrrreddgssnerrrppp..p.p neurips ceteiippsccpppppepe cc/paper files//paper/202//EE1el
pdf
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283 17297, 2020b.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEFE/CVF international conference on compuler vision, pp. 16259- 16268, 2021.
Xiaojin Zhu, Zoubin Ghahramani, and John Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In Proceedings of the Twentieth International Conference on International Conference on Machine Learning, ICML '03, pp. 912-919. AAAI Press, 2003. ISBN 1577351894.
21

---

Published in Transactions on Machine Learning Research (04/2024)
A Transformers as general sassage-passingg networks
Transformer block
Let L be the number of layers, H the number of attention heads, N the maximum
sequence length, and d the token feature dimension at the output of the transformer block H A Transformer
is defined by the iterative composition of transformer blocks sharing the following form: T(x) = -()(z)=x).
Scaled dot-product self-attention Scaled-doo-product attention updates embeddings through a weighted pooling of bosen-wise features pooled across the entire input sequence. Let us first compute (query, key, value) pairs for each token:
Q = =X = V= XWV,
with We E Rdxd, Wx E Rdxd and Wy E Rdxd, Token i-th's embodding is updated using the following formula:
X = (K,, = E K,),
JE[L,N]
In scaled-doo-product attention, the interaction scores S(,,)) are computed as follows:
S(Q,, K2) exp(Q;KT)
S
with [S:] the normalization factor associated with token i:
|S|| = Σ exp(Q.K)))
JELL.N
Multihcaded attention Multiheaded self-attention parallelizes the computation of attention into various subspaces of the latent space. We first split each input into different subspaces of latent dimension de de for queries and keys and di = 77
H
d
Amultiheaded (X) [X" ..... th H
where th is the output of the self-attention mechanism for head h [1, H]:
Asinglehead (X) -Aw wh. (X),
with W e mddd w e gaxdt, W& E gdxd and Wo e Rd.xd
Message passing neural networks MPNNs iteratively apply a local message passing equation updating node's hidden state mf+1 based on its neighbours' embeddings. This summarized representation of its local context is then fused with its previous embedding X; in order to compute its updated embedding X A In the limit of infinitely many layers, soft allention can be understood as simple message-passll passing mechanism deprived of the locality inductive bias (Kim et al. 2022b) (the aggregating neighbourhood of token is the entire soquence/token set itself N(i) = 0, n - 1))
[+1 = =3 (fx(X, Xx..., x(+1= ((( [1it,x,)) ,x)
Most efficient transformer models employing locality-sensitivn heuristics can be adequately framed under the simple message-passing framework outlined above.
For the sake of simplicity, omit the layer and head indices and the extension to the multi-layer multi-head being forward
For that purpose, a gated mechanism is often employed order to adaptively filter past information based on its relevance the task hand
22

---

Published in Transactions on Machine Learning Research (04/2024)
Transformers for point clouds The hicrarchical structure has existed in point cloud models since Point- Net++ (Qi et al. 2017b), which significantly improved upon its non-hierarchical version predecessor NointNet (Qi et al. 2017a). However, the attention operation was only successfully integrated into this dataset in Point Cloud Transformer (Guo et al., 2021). This work applicd self-attention to all points in the point cloud and gained outstanding results on object classification. Afterward, Point. Transformer (Zhao et al. 2021) leveraged the hicrarchy with local attention to enhance the performance on the segmentation prob- lems on the point cloud. Following that, Stratified Point Transformer (Lai et al., 2022) adapted voxels in order to estimate local regions for attention and positional encoding, further boosting the performance of the Transformer model in point cloud semantic segmentation. Meanwhile, another paper named Fast Point Transformer Park et al. 2022) tries to optimize the attention function and proposes a new attention-based model that is 129 times faster than Point Transformer (Zhao et al. (021) in semantic regmentation. In our work, we not only apply a new hierarchical scheme built through voxelization but also propose a new way aggregate information across the levels of the hicrarchy.
B Proofs of Computational Complexity
The total number of nodes (both true and virtual) in the tree is
A-1 |7| = Σ M Σ
A-1
n x
= n x
+ O(A)
b
1
A=
λ=
k 1
+ O
k-
(k 1)
k
(5)
1c
Thus, |7| S As, with A = k-I computational complexity of Sequoia's efficient attention scheme, both in terms of runtime and memory
k 2 1 being the dilation factor of the tree. Let us now examine the
Czequoia (n) Cobildren (n) + Cuiblings (n) Cancestors (n)
Cehildren (n)
Cobildren (n, A),
Caiblings (n)
Cabblings (n, A);
Cancestorss (n) =
Cancestors (n, X)
2=
E [0, 1] Cabildren (n, A) O(ITA k)
AA [0, A Cabblings (n, A) O k)
AAA 0, 1] Cancestors (n, A) O(ITA (A ())
O(IIa A)
Using the upper bound the number of nodes in the tree constructed by Sequoia, we get
23

---

Published in Transactions on Machine Learning Research (04/2024)
Cchildren (n) =0
A1 x nx)nxxd
Canoestors (n) = 0
Σ TT ^ [(S)xAAX4
For Caiblings (n), obtain similar bound as for Cchildren (n). We finally derive the following class of quasi- linear computational complexity for our proposed attention model
k
+ A
Csequoia (n) 0
n x
2 x
k-
= =0 (k log(n))
Indeed, the tree depth is such that A this upper bound on (zequosa(1) does not depend on the mode of computation of attention inside given transformer block (bottom-up / top-down parallel), and that the total complexity of Sequoia's 's attention scheme is dominated by the attention computed at the first level of the tred
log(a) [k]
log(n)
+ 1 = O (log(n)). In particular, we notice that
log(k)
C Visualization
In this section, extract the feature of nodes in different levels of the tree including leaf nodes, nodes from the low-level, nodes from the high-level, and root nodes. Then, we use t-SNE (van der Maaten & Hinton 2008) to reduce the features' dimension into and visualize these features in Figures observe a clear clustering pattern from these figures that suggests our hierarchical method has learned useefu and meaningful representation.
5 6 and 7 We
color
10
11
13
-20
16
18
-0
Figure 4: Visualization of the feature of root nodes in Shape Classification task.
3precisely, siblings attention the first. level and, extent, ancestors attention on the first. level.
24

---

Published in Transactions on Machine Learning Research (04/2024)
color
40
20
10
12
-20
13
15
-40
18
Figure 5: Visualization of the feature of nodes the high level of the tree in Shape Classification task.
color
20
10
12
-20
13
15
16
-40
18
Figure 6: Visualization of the feature of nodes the low level of the tree in Shape Classification task.
25

---

Published in Transactions on Machine Learning Research (04/2024)
color
100
1.0
2.0
50
-100
Figure 7: Visualization of the feature of leave nodes Part Segmentation task.
26


Reviews:
[{"review_id": "7tQvS7HuZQ", "reviewer": "Reviewer_7yAC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nExtensive experiments on a large number of datasets and tasks is a key strength of this contribution. This covers domains ranging from images to point clouds to chemistry. Ablation experiments evaluate the impact of varying hyper-parameters.\n\nComputational requirements are measured both in terms of memory footprint and inference wall-clock time. This is important as just measuring FLOPs can be misleading as some sparse operations tend to be very slow on modern accelerators.\n\n------------------------------------------------------------------------------------------------------------------------------\n### Weaknesses\n\nThe proposed method was very difficult to understand. This was in part due to some undefined terms and in part due to lack of a pseudo-code. The following details some suggested modifications:\n- In eq. 1 $\\mathcal{A}_{\\lambda}$ is undefined and can be confused with the variables in the attention equations on page 6.\n- In page 5, section 3.1, last line \"we keep the graph unweighted for simplicity\". Does this mean that there is no $\\varepsilon_ij$?\n- In page 7, in the paragraph \"Baselines and Implementation Details\" it defines number of sub-graphs = 32. \"number of sub-graphs\" is not defined in the text.\n- Section 3.3 was tricky and I think I still do not understand that part. Could the top-down and bottom-up functions be explained using pseudo-code? The same would be very helpful for the cluster learning block and would complement the figures provided.\n- Citations could be separated using () to make the text more readable.\n\nThe discussion of results is not very precise when commenting on individual tables in the text. In particular,\n- In page 8, section 4.1, \"we believe that a learnable hierarchy is a main factor that boosts the performance\" is not substantially supported by the table. The proposed method could have the top-down and bottom-up processing as the main factor, for example.\n- In page 10, section 4.3, \"Sequoia is a lightweight attention-based model with competitive results\". The results in Table 8 are unfortunately only better than PointNet but worse than all other methods. This needs to be clarified in the text.\n\n----------------------------------------------------------------------------------------------------------------\n\n### Technical concerns\n\n- In section 4.5, is the number of clusters being set per example in the experiments?\n- Sparse operations are necessary to realize the attention operations discussed in the manuscript. These normally hurt runtime on GPUs. There is no discussion about this. Did the authors find an efficient implementation?\n- The hierarchy is different per example, which means the computation graph has different tensor shapes per example. How does batch computation work in this situation?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript proposes a novel architecture that extends self attention transformers by imbuing them with an adaptive hierarchy and redefining the dot product attention operation based on the graph structure of this tree hierarchy. The result is a module that is more efficient over very long sequences and that has inductive biases based on the neighborhood structure of this tree. Lastly, the proposed module combines top-down and bottom-up interactions allowing for more expressive computation.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "A discussion about the above concerns would be helpful.", "broader_impact_concerns": "There is currently no section about broader impact concerns. I suspect that the hierarchy output by the Cluster Learning Block can help interpret model results and some discussion around this would be beneficial in a broader impact concerns section.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aitCA3Hi0K", "reviewer": "Reviewer_y1CR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. The authors propose a sensible architecture, Sequoia, for learning hierarchical self-attention.\n\n2. The experiments across many different tasks and datasets signify the usefulness of Sequoia.\n\nWeaknesses: \n\n1. The paper is missing interpretations on concrete examples. The visualizations in the appendix are not very informative because they don't delve into a connection between the hierarchies that your model learns, and the semantics of the dataset. \n\n2. I believe the benchmarking time for training is omitted from discussion in the paper, but the authors should discuss how it compares with the baselines.\n\n3. The contributions from each type of attention (described in Section 3.2) are not ablated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a new architecture for efficient self-attention by learning a hierarchy with a message passing network and Gumbel-softmax-based clustering at each level of the architecture. The claimed benefits by the authors are coming from the fact that the Gumbel-softmax allows for reducing inference latency, the incorporation of bottom-up, top a-down and intra attention contributions, as well as the log-complexity arising from the k-ary tree structure.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The second paragraph of the introduction misses citations. When you say that your \"initial motivation coincides with a similar idea\" you should cite what you mean.\n\nPlease use \\citep as suggested by the citation style. For example, when citing Reformer, you should you \\citep instead of \\citet.\n\nI did not understand claim 1 in section 2. What do you mean that these methods cannot be used on other modalities? Can you give a more concrete example to understand it?\n\nCan you give more details (maybe in the caption) about the latency benchmarking in Table 2? Question also extends to the rest of the tables.\n\nTables 5-7 are missing any parameter count and latency measurements. Why? Please add them where you can.\n\nYou should also include SOTA (for reference) on the Long Range Arena in Table 5. Likewise for Table 6.\n\nIn Table 10, why does a random hierarchy for 32 clusters perform the best?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "167LbIrLnQ", "reviewer": "Reviewer_u9PB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths\nStrong results across a variety of popular benchmarks. Efficient transformers are a hot topic that many readers will be interested in.\n\nRelated work is presented well, and the technical contribution of the paper is well motivated.\n\n**Weaknesses\n\nMy 'requested changes' below raises a number of key questions/issues I have.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose an extension of the transformer architecture to graphs, where multiple layers of attention are performed within different levels of clustering of the nodes of the graph. This enables information to be pooled at various resolutions while avoiding the quadratic complexity of a standard transformer. The clustering is done by a message-passing neural network that is trained end-to-end for the task.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My first reaction when looking at the results is to be quite skeptical. How can you have so many significant digits in your analysis? How is this statistically justifiable? Also, where do the error bars come from? Similarly, how are the reported timing numbers fair? Was everything implemented in a common software framework, run on the same hardware, and given a similar level of attention to optimizing the code?\n\nThe paper would benefit from a formal problem definition. For example, do nodes have features but not edges? How would your approach be updated to accommodate edge features? Relatedly, I was confused in sec 3. where things were described in terms of tokens instead of nodes+edges.\n\nbig-O analysis of the compute requirements for your modeling approach are insufficient in the modern where everything is vectorized. For example, many mpnn implementations assume a fully-connected graph, since this is easier to vectorize than message passing on a sparsely-connected graph. Your claims about nlogn complexity are inadequate as it does mention the practicality of achieving this complexity on modern hardware. Can you discuss the practicality, given that the structure of the graph is dynamic?\n\nThe concept of 'input size'  was poorly defined. Is the idea that your method has complexity that is linear in the number of edges? The stated complexity is nlogn; is n the number of edges? Are you assuming that the number of edges is << num_nodes ** 2? Otherwise, the mpnn stage is quadratic in the number of nodes, just like a normal transformer.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses an important and timely question of interest. All reviewers agreed that the paper is of interest to the TMLR audience, and were generally satisfied with the breadth of the experimental evaluation, along with the inclusion of runtime comparisons. Although reviewers would like to have seen more analysis done on exactly how and why the learned/adaptive hierarchies cluster+organize nodes to improve performance, the method is natural, and it is intuitive why it might work as well as it does.

The biggest reviewer concerns were around the clarity of the presentation. In their revision, alongside new experimental results, the authors added clarifications including updated notation and pseudocode for the core components of the method.

To complete this process, I would like to see the authors add pseudocode for the entire model/method, i.e. that calls the newly added Algorithms 1, 2, and 3 as sub-routines. This should constitute a "minor revision". For example, the following paragraph of the paper describes an implementation of the approach:

> "In graph-level classification, there are 1 layer of message passing to refine the feature before constructing the hierarchy, while in node-level segmentation, we apply four message-passing layers in this process. Thereafter we employ a combination of message-passing, hierarchical attention and residual layers once at each level of the tree to aggregate the global and local information. Additionally, we project the positional encoding (e.g., Laplacian (Dwivedi & Bresson, 2020b) or random walk (Dwivedi et al., 2022a)) to the same dimension of features (d = 96) and add them into the features."

The details of how everything exactly fits together here may be clear/implied to someone closely familiar with s-o-t-a of GNNs, but a more formal description of the implementation, e.g., with pseudo-code could be crucial for future researchers attempting to replicate or build on the work.

Reviewers also asked for details on how the method is efficiently run in practice. In their rebuttal, the authors discussed how they used Pytorch Geometric, including how experimental comparisons are fair w.r.t. the GPU hardware used. These details should be added to the paper, at least in the Appendix. 

Beyond this, there are some aesthetic issues that should be corrected in the revision, including the following, some of which were brought up by reviewers:
- inline citation format: use a format that includes parentheses around the citations. This issue is particular evident in the Introduction
- Some tables (2, 3, 4, and 5) are over-wide with the new columns added during the rebuttal. Please modify these so they do not go beyond the text width.
- "Baselines and Implementation Details". Should this just be "Implementation Details"? I don't see any baselines described here.
- "eq. (1)" -> "Eq. 1" and similarly where other Equations are referred to in the text.
- "Pooling eq. (1)" -> "Pooling (Eq. 1)", etc.
- "bottom-up blocks 2" -> "bottom-up blocks (Alg. 2)" and the same where Algorithms 1 and 3 are referred to in the text.
- "we would like to report" -> "we report"
- "there are 1 layer of message passing" -> "there is one layer of message passing"

I encourage the authors to check for other issues of this kind.
Decision: Accept (poster)