Title: Unitary Convolutions for Learning on Graphs and Groups
Venue: NeurIPS 2024
Accepted: yes
Keywords: graph neural networks

geometric deep learning

learning stability

unitary

orthogonal
Abstract: Data with geometric structure is ubiquitous in machine learning often arising from fundamental symmetries in a domain, such as permutation-invariance in graphs and translation-invariance in images. Group-convolutional architectures, which encode symmetries as inductive bias, have shown great success in applications, but can suffer from instabilities as their depth increases and often struggle to learn long range dependencies in data. For instance, graph neural networks experience instability due to the convergence of node representations (over-smoothing), which can occur after only a few iterations of message-passing, reducing their effectiveness in downstream tasks. Here, we propose and study unitary group convolutions, which allow for deeper networks that are more stable during training. The main focus of the paper are graph neural networks, where we show that unitary graph convolutions provably avoid over-smoothing. Our experimental results confirm that unitary graph convolutional networks achieve competitive performance on benchmark datasets compared to state-of-the-art graph neural networks. We complement our analysis of the graph domain with the study of general unitary convolutions and analyze their role in enhancing stability in general group convolutional architectures.

Unitary convolutions for learning on graphs and groups

Bobak T. Kiani ∗

Data with geometric structure is ubiquitous in machine learning often arising from fundamental symmetries in a domain, such as permutation-invariance in graphs and translation-invariance in images. Group-convolutional architectures, which encode symmetries as inductive bias, have shown great success in applications, but can suffer from instabilities as their depth increases and often struggle to learn long range dependencies in data. For instance, graph neural networks experience instability due to the convergence of node representations (over-smoothing), which can occur after only a few iterations of message-passing, reducing their effectiveness in downstream tasks. Here, we propose and study unitary group convolutions , which allow for deeper networks that are more stable during training. The main focus of the paper are graph neural networks, where we show that unitary graph convolutions provably avoid over-smoothing. Our experimental results confirm that unitary graph convolutional networks achieve competitive performance on benchmark datasets compared to state-of-the-art graph neural networks. We complement our analysis of the graph domain with the study of general unitary convolutions and analyze their role in enhancing stability in general group convolutional architectures. 4

1 Introduction

In recent years, the design of specialized machine learning architectures for structured data has received a surge of interest. Of particular interest are architectures for data domains with inherent symmetries, such as permutation-invariance in graphs and sets, translation-invariance in images, and other symmetries that arise from fundamental laws of physics in scientific data.

Group-convolutional architectures allow for explicitly encoding symmetries as inductive biases, which has led to performance gains in scientific applications [ESM23, WWY20]; theoretical studies have analyzed the impact of geometric inductive biases on the complexity of the architecture [MMM21, BVB21, KLL + 24]. Graph Neural Networks are among the most popular architectures for graph machine learning and have found impactful applications in a wide range of disciplines, including in chemistry [GRK + 21], drug discovery [ZAL18], particle physics [SBV20], and recommender systems [WSZ + 22]. However, despite these successes, several limitations remain. A notable difficulty is the design of stable, deep architectures. Many of the aforementioned applications require accurate learning of long-range dependencies in the data, which necessitates deeper networks. However, it has been widely observed that group-convolutional networks suffer from instabilities as their depths increases. On graph domains, these instabilities have been studied extensively in recent years, notably in the form of over-smoothing effects [RBM23], which characterizes the fast convergence of the representations of nearby nodes with depth. This effect can often be observed after only a few iterations of message-passing (i.e., small number of layers) and can significantly decrease the utility of the learned representations in downstream tasks. While interventions for mitigating over-smoothing have been proposed, including targeted perturbations of the input graph's connectivity (rewiring) and skip connections, a more principled architectural approach with theoretical guarantees is still

lacking. Similar effects, such as exploding or vanishing gradients, have also been studied in more general group-convolutional architectures, specifically in CNNs [TK21, SF21, LJW + 19], for which architectural interventions (e.g., skip connections) have been proposed.

In this work, we take a different route. Inspired by a long line of work studying unitary recurrent neural networks [ASB16, HSL16, LCMR19, KBLL22], we propose to replace the standard group convolution operator with a unitary group convolution. By construction, the unitarity ensures that the linear transformations are norm-preserving and invertible, which can significantly enhance the stability of the network and avoid convergence of representations to a fixed point as its depth increases. We introduce two unitary graph convolution operators, which vary in the way the message passing and feature transformation are parameterized. We then generalize this approach to cover more general group-convolutional architectures.

Our theoretical analysis of the proposed unitary graph convolutions shows that they enhance stability and prevent over-smoothing effects that decrease the performance of their vanilla counterparts. We further describe how generalized unitary convolutions avoid vanishing and exploding gradients, enhancing the stability of group-convolutional architectures without additional interventions, such as residual connections or batch normalization.

1.1 Related work

Wenowprovideabrief background into work that motivated and inspired this current study, deferring a more complete discussion to App. A. Unitary matrices have a long history of application in neural networks, specifically related to improving stability for deep networks [SMG13, PMB13] enhancing the learning of long-range dependencies in data [BSF94, Hoc91, SMG13]. [ASB16, JSD + 17, HSL16] implemented unitary matrices in recurrent neural networks to address issues with the challenge of vanishing and exploding gradients inherent to learning long sequences of data in RNNs [BSF94, LJH15]. These original algorithms were later improved to be more expressive while still being efficient to implement in practice [HWY18, LCMR19, KBLL22]. For graph convolution, [HSTW20] discuss applications of the exponential map to linear convolutions; here, we use this same exponential map to explicitly apply unitary operators parameterized in the Lie algebra. For image data, various works [SGL18, LHA + 19, TK21, SF21, KBLL22] design and analyze variants of orthogonal or unitary convolution used in CNN layers. These can be viewed as a particular instance of the group convolution we study here over the cyclic group. More recently, proposals for unitary or orthogonal message passing have shown improvements in stability and performance compared to conventional message passing approaches [GZH + 22, AEL + 24, QBY24]. However, in contrast to our work, these methods do not always implement a unitary transformation across the whole input (e.g. only applying it in the feature transformation) and in the case of [QBY24] can be computationally expensive to implement for large graphs (see App. A for more detail).

2 Background and Notation

We denote scalars, vectors, and matrices as 𝑐, w , and M respectively. Given a matrix M , its conjugate transpose and transpose are denoted M † and M ⊤ . Given two matrices, A and B , we denote their tensor product (or Kronecker product) as A ⊗ B . Given a vector w , its standard Euclidean norm is denoted ∥ w ∥ . For a matrix M , we denote its operator norm as ∥ M ∥ and Frobenius norm as ∥ M ∥ 𝐹 .

Group Theory Basics Symmetries ('invariances') describe transformations, which leave properties of data unchanged ('invariant'), and as such characterize the inherent geometric structure of the data domain. Algebraically, symmetries can be characterized as groups. We say that a group is a matrix Lie group , if it is a differentiable manifold and a subgroup of the set of invertible 𝑛 × 𝑛 matrices (see App. C). Lie groups are associated with a Lie algebra, a vector space, which is formed by its tangent space at the identity. A comprehensive introduction into Lie groups and Lie algebras can be found in [FH13, Hal15]. Throughout this work we will encounter the 𝑛 -dimensional orthogonal 𝑂 ( 𝑛 ) and unitary 𝑈 ( 𝑛 ) Lie groups, which are defined as

Lie algebras of 𝑂 ( 𝑛 ) and 𝑈 ( 𝑛 ) are the set of skew symmetric 𝔬 ( 𝑛 ) and skew Hermitian 𝔲 ( 𝑛 ) matrices respectively, i.e.,

Given a matrix M ∈ 𝔬 ( 𝑛 ) (or 𝔲 ( 𝑛 ) ), the matrix exponential maps the matrix to an element of the Lie group exp ( M ) ∈ 𝑂 ( 𝑛 ) (or 𝑈 ( 𝑛 ) ). More details on unitary parametrizations can be found in App. C.2.

Graph Neural Networks We denote graphs by G = ( 𝑉, 𝐸 ) where 𝑉 and 𝐸 denote the set of nodes and edges respectively. For a graph on 𝑛 nodes, unless otherwise specified, we let 𝑉 = { 1 , . . . , 𝑛 } index the nodes and denote the adjacency matrix by A ∈ R 𝑛 × 𝑛 and node features x ∈ R 𝑛 × 𝑑 . We also often use D ∈ R 𝑛 × 𝑛 to denote the diagonal degree matrix where diagonal entry 𝑖 records the degree of node 𝑖 . The normalized adjacency matrix is defined as e A = D -1 / 2 AD -1 / 2 . Given a node feature matrix X ∈ R 𝑛 × 𝑑 in where row 𝑖 denotes the 𝑑 in -dimensional feature vector of node 𝑖 , graph convolution operators take the general form [KW16, ZK20]

where W 0 , . . . W 𝑘 ∈ R 𝑑 in × 𝑑 out are trainable parameters. For ease of presentation, we will often omit the adjacency matrix as an explicit input to the operation. Often, only a single 'message passing' step is included and the operation takes the simple form 𝑓 conv ( X ) = AXW . Eq. (3) is equivariant under any permutation matrix P 𝜋 ∈ R 𝑛 × 𝑛 5 since

We aim to parameterize a particular subset of these operations which preserve unitarity properties.

Group-Convolutional Neural Networks In general, a linear convolution operation conv 𝐺 : C 𝑛 × 𝑐 → C 𝑛 × 𝑐 takes the form of a weighted sum over linear transformations that are equivariant to a given group 𝐺 . For simplicity, we assume here that we are working with finite groups though this can be generalized to other settings [KT18, CGKW18, CW16]. Given an input X ∈ C 𝑛 × 𝑐 consisting of 𝑐 channels in a vector space of dimension 𝑛 , we study convolutions of the form

where T 1 , . . . , T 𝑚 ∈ C 𝑛 × 𝑛 are linear operators equivariant to the group 𝐺 and W 1 , . . . , W 𝑚 ∈ C 𝑐 × 𝑐 are parameterized weight matrices. The graph setting is recovered by setting T 𝑘 = A 𝑘 . Similarly, for cyclic convolution as in conventional CNNs, one sets T 𝑘 to be the circulant matrices sending basis vector T 𝑘 e 𝑖 = e 𝑖 + 𝑘 where indexing is taken mod 𝑛 .

3 Unitary Group Convolutions

We first describe unitary convolution for data on graphs (equivariant to permutations) which is the main focus of our study and then detail general procedures for performing unitary convolutions equivariant to general finite groups. Implementing these operations often requires special considerations to handle nonlinearities, complex numbers, initialization, etc. which we discuss in App. E.

3.1 Unitary graph convolution

We introduce two variants of unitary graph convolution, which we denote as Separable unitary convolution (short UniConv ) and Lie orthogonal/unitary convolution (short Lie UniConv ). UniConv is a simple adjustment to standard message passing and treats linear transformations over nodes and features separately. Lie UniConv, in contrast, parameterizes operations in the Lie algebra of the orthogonal/unitary groups. This operation is fully unitary, but does not have the tensor product nature of the separable UniConv.

By introducing complex numbers, we can enforce unitarity separately in the message passing and feature transformation.

0

20

40

60

80

Node Index

0.0

0.2

0.4

Magnitude

Standard Message Passing

L=0

L=2

L=5

0

20

40

60

80

Node Index

Unitary Message Passing

L=0

L=2

L=5

Definition 1 (Separable unitary graph convolution (UniConv)) . Given an undirected graph G over 𝑛 nodes with adjacency matrix A ∈ R 𝑛 × 𝑛 , separable unitary graph convolution (UniConv) 𝑓 Uconv : C 𝑛 × 𝑑 → C 𝑛 × 𝑑 takes the form

where U ∈ 𝑈 ( 𝑑 ) is a unitary operator and 𝑡 ∈ R controls the magnitude of the convolution.

One feature of the complexification of the adjacency matrix is that messages propogate as 'waves' as observed for example in Fig. 1. Since A is a symmetric matrix, exp ( 𝑖 A 𝑡 ) is unitary for all values of 𝑡 ∈ R and corresponds to vanilla message passing up to first order: exp ( 𝑖 A 𝑡 ) ≈ I + 𝑖 A 𝑡 + 𝑂 ( 𝑡 2 ) .

Remark 2. We observe that performance on real-world tasks is usually improved when enforcing unitarity in the node message passing where oversmoothing occurs, but not necessarily when enforcing unitarity in the feature transformation U . Thus, one can choose to leave U in Eq. (6) as a fully parameterized (unconstrained) matrix as we often do in our experiments.

More generally, one can parameterize the operation in the Lie algebra by first forming a skew Hermitian convolution operation 𝑔 conv : C 𝑛 × 𝑑 → C 𝑛 × 𝑑 and then applying the exponential map. This approach has the benefit that it can be fully implemented using real numbers to obtain an orthogonal operator by enforcing constraints in the real part of the weight matrix W only.

Definition 3 (Lie orthogonal/unitary graph convolution (Lie UniConv)) . Given an undirected graph G over 𝑛 nodes with adjacency matrix A ∈ R 𝑛 × 𝑛 , Lie unitary/orthogonal graph convolution (OrthoConv) 𝑓 Uconv : C 𝑛 × 𝑑 → C 𝑛 × 𝑑 takes the form

The operation of 𝑔 conv can be represented as a vector-matrix operation vec ( 𝑔 conv ( X )) = A ⊗ W ⊤ vec ( 𝑋 ) where A ⊗ W ⊤ belongs in the Lie algebra. If W is real-valued, the above returns an orthogonal map since the exponential of a real-valued matrix is real-valued.

Implementing the exponential map The exponential map in Definitions 1 and 3 can be performed using accurate approximations with typically constant factor overhead in runtime. We use the simple 𝐾 -th order Taylor approximation

For experiments, we find that setting 𝐾 = 10 suffices in all cases as the error exponentially decreases with 𝐾 . Various other accurate and efficient approximations exist as detailed in App. C.2. We also

refer the reader to App. E for other implementation details associated to handling complex numbers, initialization, etc.

3.2 Generalized unitary convolutions

In the more general setting, we are concerned with parameterizing unitary operations of the form

where T 1 , . . . , T 𝑚 ∈ C 𝑛 × 𝑛 are linear operators equivariant to the group 𝐺 and W 1 , . . . , W 𝑚 ∈ C 𝑐 × 𝑐 are parameterized weight matrices (e.g., set T 𝑘 = A 𝑘 -1 to recover graph convolution). One can enforce and parameterize unitary convolutions in Eq. (9) in the Lie algebra basis or in the Fourier domain as detailed in App. D.

Algorithm 1 Unitary map from Lie algebra

Input: equivariant linear operator L ∈ C 𝑛 → C 𝑛

Input:

vector

x

∈

C

𝑛

1: e L = 1 2 ( L - L † ) (skew symmetrize operator)

2: return exp ( e L )( x ) (or approximation thereof)

In the Lie algebraic setting (Algorithm 1), one explicitly parameterizes operators in the Lie algebra or orthogonally projects arbitrary linear operators onto this basis by the mapping X ↦→( X -X † )/ 2. This parameterization is particularly simple to implement (it is a linear basis) and unitary operators are subsequently implemented by applying the exponential map. This setting covers previous implementations of unitary RNNs and CNNs [LCMR19, SF21] and is detailed in Sec. 3.1 for GNNs.

Example 1 (Convolution on regular representation (Lie algebra)) . Given a group 𝐺 and vector space V of dimension | 𝐺 | with basis { e 𝑔 : 𝑔 ∈ 𝐺 } , then the left action T 𝑔 (right action R 𝑔 ) of any 𝑔 ∈ 𝐺 is a permutation T 𝑔 e ℎ = e 𝑔 -1 ℎ ( R 𝑔 e ℎ = e ℎ𝑔 ). Let x ∈ C | 𝐺 | be the vectorized form of an input function 𝑥 : 𝐺 → C and 𝑚 : 𝐺 → C the filter for convolution 6

  Parameterizing operations on the Lie algebra simply requires that 𝑚 ( 𝑔 ) = 𝑚 ( 𝑔 -1 ) ∗ since R -1 𝑔 = R † 𝑔 .

An example implementation of the above for a toy learning task on the dihedral group is in App. F.3. One can also generally implement convolutions in the (block diagonal) Fourier basis of the graph or group (see App. D and Algorithm 3). Here, one employs a Fourier operator which block diagonalizes the input into its irreducible representations or some spectral representation. Fourier representations often have the advantage of being faster to implement due to efficient Fourier transforms. Since we do not use this in our experiments, we defer the details to App. D.

4 Properties and theoretical guarantees

Unitary operators are now well studied in the context of neural networks and have various properties that are useful in naturally enhancing stability and performance of learning architectures [ASB16, SMG13,LCMR19,JSD + 17, KBLL22]. These properties and their theoretical guarantees are outlined here. We defer all proofs to App. B, many of which follow immediately from the definition of unitarity.

Throughout, we will assume that convolution operators act on a vector space V and are built from a basis of linear operators that is equivariant to input and output representation 𝜌 ( 𝑔 ) of a group 𝐺 . We set input/output representations to be equal so that the exponential map of an equivariant operator is itself equivariant.

Fact 1 (Basic properties) . Any unitary convolution 𝑓 Uconv : V → V built from Algorithms 1 and 3 meets the basic properties:

A simple corollary of the above isometry property leveraged in prior work on unitary CNNs [TK21, SGL18] is that unitary matrices naturally provide robustness guarantees and a provable means to bound the effects of adversarial perturbations (see Corollary 9 in App. B). For graphs, we note that the properties above are generally impossible to obtain with graph convolution operations that perform a single message passing step as we show below.

Proposition 4. Let 𝑓 conv : R 𝑛 × 𝑑 → R 𝑛 × 𝑑 be a graph convolution layer of the form

where W 0 , W 1 ∈ R 𝑑 × 𝑑 are parameterized matrices. The linear map 𝑓 (· , A ) : R 𝑛 × 𝑑 → R 𝑛 × 𝑑 is orthogonal for all adjacency matrices A of undirected graphs only if W 1 = 0 and W 0 ∈ 𝑂 ( 𝑑 ) is orthogonal. Furthermore, denoting J A ∈ R 𝑛𝑑 × 𝑛𝑑 as the Jacobian matrix of the map 𝑓 conv (· , A ) , for any choice of W 0 , W 1 , there always exists a normalized adjacency matrix ˆ A such that

where ∥ M ∥ is the operator norm of matrix M .

The above shows that one must apply higher powers of A as in the exponential map to achieve a linear operator that is close to orthogonal.

Oversmoothing During the training of GNNs we often observe that the features of neighboring nodes become more similar as the depth of the networks (i.e., the number of message-passing iterations) increases. This 'oversmoothing' phenomenon has a strong connection to the spectral properties of graphs where convergence of a function on the graph is measured through the Dirichlet form 7 or its normalized variant also termed the Rayleigh quotient.

Definition 5 (Rayleigh quotient [Chu97]) . Given an undirected graph G = ( 𝑉, 𝐸 ) on | 𝑉 | = 𝑛 nodes with adjacency matrix A ∈ { 0 , 1 } 𝑛 × 𝑛 , let D ∈ R 𝑛 × 𝑛 be a diagonal matrix where the 𝑖 -th entry D 𝑖𝑖 = 𝑑 𝑖 and 𝑑 𝑖 is the degree of node 𝑖 . Let 𝑓 : 𝑉 → C 𝑑 be a function from nodes to features. Then the Rayleigh quotient 𝑅 G ( 𝑓 ) is equal to

where e A = D -1 / 2 AD -1 / 2 is the normalized adjacency matrix and X ∈ C 𝑛 × 𝑑 is a matrix with the 𝑖 -th row set to feature vector 𝑓 ( 𝑖 ) . We will at times abuse notation and let X be an input to 𝑅 G ( X ) .

Given an undirected graph G on 𝑛 nodes with normalized adjacency matrix e A = D -1 / 2 AD -1 / 2 , we compare the Rayleigh quotient of normalized vanilla and unitary convolution. First, it is straightforward to show that the Rayleigh quotient is invariant to unitary transformations giving a proof that unitary graph convolution avoids oversmoothing.

Proposition 6 (Invariance of Rayleigh quotient) . Given an undirected graph G on 𝑛 nodes with normalized adjacency matrix e A = D -1 / 2 AD -1 / 2 , the Rayleigh quotient 𝑅 G ( X ) = 𝑅 G ( 𝑓 Uconv ( X )) is invariant under normalized unitary or orthogonal graph convolution (see Definitions 1 and 3).

In contrast, oversmoothing commonly occurs with vanilla graph convolution and has been proven to occur in a variety of settings [RBM23, CW20, BDGC + 22, Ker22]. To illustrate this, we exhibit a simple setting below commonly found at initialization where the parameterized matrix is set to be an orthogonal matrix and input features are random. Here, the magnitude of oversmoothing concentrates around its average and grows with the value of Tr ( e A 3 ) which corresponds to the (weighted) number of triangles in the graph.

Proposition 7. Given a simple undirected graph G on 𝑛 nodes with normalized adjacency matrix e A = D -1 / 2 AD -1 / 2 and node degree bounded by 𝐷 , let X ∈ R 𝑛 × 𝑑 have rows drawn i.i.d. from the uniform distribution on the hypersphere in dimension 𝑑 . Let 𝑓 conv ( X ) = e AXW denote convolution with orthogonal feature transformation matrix W ∈ 𝑂 ( 𝑑 ) . Then, the event below holds with probability 1 -exp (-Ω ( √ 𝑛 )) :

Vanishing/Exploding gradients A commonly observed issue in training deep neural networks, especially RNNs with evolving hidden states, is that gradients can exponentially grow or vanish with depth [HS97, GBC16]. In fact, one of the original motivations for using unitary matrices in RNNs is to directly avoid this vanishing/exploding gradient problem [ASB16, LCMR19, JSD + 17, HSL16]. From a theoretical view, prior work has shown that carefully initialized layers have Jacobians that meetvariants of the dynamical isometry property commonly studied in the mean field theory literature characterizing the growth/decay over layers of the network [SMG13, XBSD + 18, PSG18]. We analyze a version of this property here and discuss it in the context of our work.

Definition 8 (Dynamical isometry) . Given functions 𝑓 1 , . . . , 𝑓 𝐿 : R 𝑛 → R 𝑛 , let 𝐹 𝑖 = 𝑓 𝑖 ◦ · · · ◦ 𝑓 1 . Let J 𝐹 𝑖 ( x ) be the Jacobian matrix of 𝐹 𝑖 at x ∈ R 𝑛 8 . The function 𝐹 𝐿 = 𝑓 𝐿 ◦ · · · ◦ 𝑓 1 is dynamically isometric up to 𝜖 at x ∈ R 𝑛 if there exists orthogonal matrix V ∈ 𝑂 ( 𝑛 ) such that GLYPH<13> GLYPH<13> ˛ 𝐿 𝑖 = 1 J 𝐹 𝑖 ( x ) -V GLYPH<13> GLYPH<13> ≤ 𝜖 where ∥ · ∥ denotes the operator norm.

Network layers that meet the dynamical isometry property generally avoid vanishing/exploding gradients. The particular form we analyze is stricter than those studied in the mean field and Gaussian process literature which analyze the distribution of singular values over the randomness of the weights [PSG18, XBSD + 18]. Unitary convolution layers followed by isometric activations are examples of dynamical isometries that hold throughout training as exemplified below.

Example 2. Compositions of the layer GroupSort ( 𝑓 Uconv ( x )) consisting of the unitary convolution layer (Definition 1) followed by the Group Sort activation (Eq. (61)) are perfectly dynamically isometric ( 𝜖 = 0) at all x ∈ C 𝑛 9 .

5 Experiments

Our experimental results here show that unitary/orthogonal variants of graph convolutional networks perform competitively on various graph learning tasks. Due to space constraints, we present experiments on additional datasets and architectures in App. F. This includes experiments on TUDataset [MKB + 20] and an instance of unitary group convolutional networks on the dihedral group where the goal is to learn distances between pairs of elements in the dihedral group. Training procedures and hyperparameters are reported in App. G. Reported results in tables are over the mean plus/minus standard deviation.

Toy model: graph distance To analyze the ability of our unitary GNN to learn long-range dependencies, we consider a toy dataset where the aim is to learn the distance between two indicated nodes on a large graph connected as a ring. This task is inspired by similar toy models on ring graphs where prior work has shown that message passing architectures fail to learn long-range dependencies between distant nodes [DGGB + 23]. The particular dataset we analyze consists of a training set of 𝑁 = 1000 graphs on 𝑛 = 100 nodes where each graph is connected as a ring (see Fig. 2a). Node features x 𝑖 ∈ R are a single number set to zero for all but two randomly chosen nodes whose features are set to one. The goal is to predict the distance between these two randomly chosen nodes. For a graph of 𝑛 nodes, conventional message passing architectures require at least 𝑛 / 2 sequential messages to fully learn this dataset. As shown in Fig. 2b, conventional message passing networks fail to learn this task whereas the unitary convolutional architecture succeeds. We refer the reader to App. F.1 for further details and results for additional architectures.

Reported performance taken from [TRRG23].

†

0

50

100

150

200

250

0.0

2.5

5.0

7.5

10.0

12.5

test MAE

Graph Attention Network

Residual GCN

Spectral Convolution

Vanilla GCN

Unitary GCN (UniConv)

Unitary GCN (Lie UniConv)

Trivial Performance

Figure 2: (a) Example datapoint on 𝑛 = 25 nodes; the target is 𝑦 = 5 (distance between red nodes). (b) Results for the ring toy model problem with 100 nodes where the unitary GCN with UniConv or Lie UniConv layers is the only message passing architecture able to learn successfully. Best performance over networks with 5, 10, and 20 layers is plotted. Other architectures typically perform best with 5 layers and only learn shorter distances (see App. F.1).

Long Range Graph Benchmark (LRGB) We consider the Peptides, Coco, and Pascal datatsets from the Long Range Graph Benchmark (LRGB) [DRG + 22]. There are two tasks associated with Peptides, a peptide function classification task (Peptides-func) and a regression task (Peptides-struct). Coco and Pascal are node classification tasks.Table 1 shows that the Unitary GCN outperforms standard message passing architectures and is competitive with other state of the art architectures as well, many of which employ global attention mechanisms. This provides evidence that the unitary GCN is nearly as effective at learning long-range signals.

Heterophilous Graph Dataset For node classification, we consider the Heterophilous Graph Dataset proposed by [PKD + 23]. The dataset contains the heterophilous graphs Roman-empire, Amazon-ratings, Minesweeper, Tolokers, and Questions, which are often considered as a benchmark for evaluating the performance of GNNs on graphs where connected nodes have dissimilar labels and features. Our results in Table 2 show that the unitary GCN outperforms the baseline message passing and graph transformer models. Given the heterophilous nature of this dataset, these findings reinforce the notion that unitary convolution enhances the ability of convolutional networks to capture long-range dependencies.

6 Discussion

In this paper we introduced unitary graph convolutions for enhancing stability in graph neural networks. We provided theoretical and empirical evidence for the effectiveness of our approach. We further introduce an extension to general groups (generalized unitary convolutions), which can be leveraged in group-convolutional architectures to enhance stability.

Reported performance taken from [PKD

†

+

23].

Limitations Perhaps the biggest challenge in working with unitary convolutions is the overhead associated with maintaining unitarity or orthogonality via approximations of the exponential map or diagonalizations in Fourier or spectral bases. For implementing unitary maps, working with complex numbers also requires different initialization and activation functions. We refer the reader to App. C.2 and E for methods to alleviate these challenges in practice. Separately, there may be target functions or problem instances where unitarity or orthogonality may not be appropriate. For example, one can envision node classification tasks where the target function is neither (approximately) invertible nor isometric. In such instances, non-unitary layers will be required to learn the task. More generally, deciding when to use unitary layers is problem-dependent. In some cases, such as applications with smaller input graphs, simple and more efficient interventions such as adding residual connections or including batch norm will likely suffice for addressing signal propagation problems.

FutureDirections In the graph domain, extensions of graph convolution to more advanced methods such as those that better incorporate edge features could widen the range of applications. Exploring hybrid models that combine unitary and non-unitary layers (e.g. global attention mechanisms) could potentially lead to more robust and versatile graph neural networks. Future work can also improve the efficiency of the parameterizations and implementations of the exponential map (see App. C.2). In a similar vein, it is likely that approximately unitary/orthogonal layers suffice in many settings to achieve the performance gains we see in our work. Methods that approximately enforce or regularize layers towards unitarity may be of interest in these instances due to their potential for improved efficiency. In this study, we mainly focused on applications to graph classification and regression tasks; however, the proposed methodology is much more general and could open up a wider range of applications to domains with more general symmetries or different data domains. For example, unitary matrices offer provable guarantees to adversarial attacks (see Corollary 9) and testing this robustness in practice on geometric data has yet to be conducted.

References

[ABKL23] Eric R Anschuetz, Andreas Bauer, Bobak T Kiani, and Seth Lloyd. Efficient classical algorithms for simulating symmetric quantum systems. Quantum , 7:1189, 2023. 18

[AEL + 24] Yassine Abbahaddou, Sofiane Ennadir, Johannes F. Lutzeyer, Michalis Vazirgiannis, and Henrik Bostr¨ om. Bounding the expected robustness of graph neural networks subject to node feature attacks. In The Twelfth International Conference on Learning Representations , 2024. 2, 18, 19

[AK22] Eric R Anschuetz and Bobak T Kiani. Quantum variational algorithms are swamped with traps. Nature Communications , 13(1):7760, 2022. 18

[ALG19] Cem Anil, James Lucas, and Roger Grosse. Sorting out lipschitz function approximation. In International Conference on Machine Learning , pages 291-301. PMLR, 2019. 27, 31

[AMH09] Awad H Al-Mohy and Nicholas J Higham. Computing the fr´ echet derivative of the matrix exponential, with an application to condition number estimation. SIAM Journal on Matrix Analysis and Applications , 30(4):1639-1657, 2009. 25

[AMH10] Awad H Al-Mohy and Nicholas J Higham. A new scaling and squaring algorithm for the matrix exponential. SIAMJournal on Matrix Analysis and Applications , 31(3):970989, 2010. 25

[ASB16] Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International Conference on Machine Learning , pages 1120-1128, 2016. 2, 5, 7, 17

[BDGC + 22] Cristian Bodnar, Francesco Di Giovanni, Benjamin Chamberlain, Pietro Lio, and Michael Bronstein. Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in gnns. Advances in Neural Information Processing Systems , 35:18527-18541, 2022. 6

[BL17] Xavier Bresson and Thomas Laurent. Residual gated graph convnets. arXiv preprint arXiv:1711.07553 , 2017. 18, 30

[BQL21] Joshua Bassey, Lijun Qian, and Xianfang Li. A survey of complex-valued neural networks. arXiv preprint arXiv:2101.12249 , 2021. 27

[BSC + 24] Ilyes Batatia, Lars Leon Schaaf, Gabor Csanyi, Christoph Ortner, and Felix Andreas Faber. Equivariant matrix function neural networks. In The Twelfth International Conference on Learning Representations , 2024. 18

[BSF94] Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks , 5(2):157-166, 1994. 2

[BVB21] Alberto Bietti, Luca Venturi, and Joan Bruna. On the sample complexity of learning under geometric stability. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 18673-18684. Curran Associates, Inc., 2021. 1

[CGKW18] Taco S Cohen, Mario Geiger, Jonas K¨ ohler, and Max Welling. Spherical cnns. arXiv preprint arXiv:1801.10130 , 2018. 3, 18

[Chu97] Fan RK Chung. Spectral graph theory , volume 92. American Mathematical Soc., 1997. 6

[CNDP + 21] Grecia Castelazo, Quynh T Nguyen, Giacomo De Palma, Dirk Englund, Seth Lloyd, and Bobak T Kiani. Quantum algorithms for group convolution, cross-correlation, and equivariant transformations. arXiv preprint arXiv:2109.11330 , 2021. 18

[CST + 21] Andrew M Childs, Yuan Su, Minh C Tran, Nathan Wiebe, and Shuchen Zhu. Theory of trotter error with commutator scaling. Physical Review X , 11(1):011020, 2021. 25

[CW16] Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning , pages 2990-2999. PMLR, 2016. 3, 18

[CW20] Chen Cai and Yusu Wang. A note on over-smoothing for graph neural networks. arXiv preprint arXiv:2006.13318 , 2020. 6, 28

[DGGB + 23] Francesco Di Giovanni, Lorenzo Giusti, Federico Barbero, Giulia Luise, Pietro Lio, and Michael M Bronstein. On over-squashing in message passing neural networks: The impact of width, depth, and topology. In International Conference on Machine Learning , pages 7865-7885. PMLR, 2023. 7

[DRG + 22] Vijay Prakash Dwivedi, Ladislav Ramp´ aˇ sek, Mikhail Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. Long range graph benchmark. In Thirtysixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022. 8, 33

[ESM23] Carlos Esteves, Jean-Jacques Slotine, and Ameesh Makadia. Scaling spherical cnns. arXiv preprint arXiv:2306.05420 , 2023. 1

[FH13] William Fulton and Joe Harris. Representation theory: a first course , volume 129. Springer Science & Business Media, 2013. 2, 20, 26

[FL19] Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds , 2019. 30, 33

[FSIW20] MarcFinzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. In International Conference on Machine Learning , pages 3165-3176. PMLR, 2020. 18

[FW24] Lukas Fesser and Melanie Weber. Mitigating over-smoothing and over-squashing using augmentations of Forman-Ricci curvature. In Proceedings of the Second Learning on Graphs Conference , volume 231 of Proceedings of Machine Learning Research , pages 19:1-19:28. PMLR, 27-30 Nov 2024. 18

[GBC16] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning . MIT press, 2016. 7

[GBH18] Octavian Ganea, Gary B´ ecigneul, and Thomas Hofmann. Hyperbolic neural networks. Advances in neural information processing systems , 31, 2018. 18

[GDBDG23] Benjamin Gutteridge, Xiaowen Dong, Michael M. Bronstein, and Francesco Di Giovanni. DRew: Dynamically rewired message passing with delay. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 12252-12267. PMLR, 23-29 Jul 2023. 18, 30

[GRK + 21] Gligorijevi´ c, Renfrew, Kosciolek, Leman Koehler, Berenberg, Vatanen, Chandler, Taylor, Fisk, Vlamakis, et al. Structure-based protein function prediction using graph convolutional networks. Nature communications , 12(1):3168, 2021. 1

[GX15] Qinghua Guo and Jiangtao Xi. Approximate message passing with unitary transformation. arXiv preprint arXiv:1504.04799 , 2015. 18

[GZH + 22] Kai Guo, Kaixiong Zhou, Xia Hu, Yu Li, Yi Chang, and Xin Wang. Orthogonal graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 3996-4004, 2022. 2, 18, 19

[Hal15] Brian Hall. Lie groups, Lie algebras, and representations: an elementary introduction , volume 222. Springer, 2015. 2, 23, 24, 25

[HHL + 23] Xiaoxin He, Bryan Hooi, Thomas Laurent, Adam Perold, Yann Lecun, and Xavier Bresson. A generalization of ViT/MLP-mixer to graphs. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 12724-12745. PMLR, 23-29 Jul 2023. 18, 30

[Hig09] Nicholas J Higham. The scaling and squaring method for the matrix exponential revisited. SIAM review , 51(4):747-764, 2009. 25

[Hoc91] Sepp Hochreiter. Untersuchungen zu dynamischen neuronalen netzen. Diploma, Technische Universit¨ at M¨ unchen , 91(1), 1991. 2

[HS97] Sepp Hochreiter and J¨ urgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997. 7, 17

[HSL16] Mikael Henaff, Arthur Szlam, and Yann LeCun. Recurrent orthogonal networks and long-memory tasks. In International Conference on Machine Learning , pages 20342042. PMLR, 2016. 2, 7, 27

[HSTW20] Emiel Hoogeboom, Victor Garcia Satorras, Jakub M Tomczak, and Max Welling. The convolution exponential and generalized sylvester flows. arXiv preprint arXiv:2006.01910 , 2020. 2

[HWY18] Kyle Helfrich, Devin Willmott, and Qiang Ye. Orthogonal recurrent neural networks with scaled cayley transform. In International Conference on Machine Learning , pages 1969-1978. PMLR, 2018. 2, 17, 25, 27

[HYL17] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017. 18

[JD15] Bo Jiang and Yu-Hong Dai. A framework of constraint preserving update schemes for optimization on stiefel manifold. Mathematical Programming , 153(2):535-575, 2015. 27

[JSD + 17] Li Jing, Yichen Shen, Tena Dubcek, John Peurifoy, Scott Skirlo, Yann LeCun, Max Tegmark, and Marin Soljaˇ ci´ c. Tunable efficient unitary neural networks (eunn) and their application to rnns. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 1733-1741. JMLR. org, 2017. 2, 5, 7, 17

[KB14] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. 31

[KBLL22] Bobak Kiani, Randall Balestriero, Yann LeCun, and Seth Lloyd. projunn: efficient method for training deep networks with unitary matrices. Advances in Neural Information Processing Systems , 35:14448-14463, 2022. 2, 5, 17, 18, 20, 26, 27

[Kel75] Joseph B Keller. Closest unitary, orthogonal and hermitian operators to a given operator. Mathematics Magazine , 48(4):192-197, 1975. 19

[Kem03] Julia Kempe. Quantum random walks: an introductory overview. Contemporary Physics , 44(4):307-327, 2003. 18

[Ker22] Nicolas Keriven. Not too little, not too much: a theoretical analysis of graph (over) smoothing. Advances in Neural Information Processing Systems , 35:2268-2281, 2022. 6

[KJ08] Alexander Kirillov Jr. An introduction to Lie groups and Lie algebras . Number 113. Cambridge University Press, 2008. 24

[KLL + 24] Bobak T Kiani, Thien Le, Hannah Lawrence, Stefanie Jegelka, and Melanie Weber. On the hardness of learning under symmetries. In International Conference on Learning Representations , 2024. 1

[KT18] Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International conference on machine learning , pages 2747-2755. PMLR, 2018. 3, 18, 20, 26

[KW16] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016. 3, 18, 28, 30

[LCMR19] Mario Lezcano-Casado and David Martınez-Rubio. Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. In International Conference on Machine Learning , pages 3794-3803. PMLR, 2019. 2, 5, 7, 17, 20, 24, 25, 27

[LFT20] Jun Li, Li Fuxin, and Sinisa Todorovic. Efficient riemannian optimization on the stiefel manifold via the cayley transform. arXiv preprint arXiv:2002.01113 , 2020. 20, 27

[LGJ + 21] Man Luo, Qinghua Guo, Ming Jin, Yonina C Eldar, Defeng Huang, and Xiangming Meng. Unitary approximate message passing for sparse bayesian learning. IEEE transactions on signal processing , 69:6023-6039, 2021. 18

[LHA + 19] Qiyang Li, Saminul Haque, Cem Anil, James Lucas, Roger B Grosse, and J¨ ornHenrik Jacobsen. Preventing gradient attenuation in lipschitz constrained convolutional networks. Advances in neural information processing systems , 32:15390-15402, 2019. 2, 17, 20

[LJH15] Quoc V Le, Navdeep Jaitly, and Geoffrey E Hinton. A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXiv:1504.00941 , 2015. 2

[LJW + 19] Shuai Li, Kui Jia, Yuxin Wen, Tongliang Liu, and Dacheng Tao. Orthogonal deep neural networks. IEEE transactions on pattern analysis and machine intelligence , 43(4):1352-1368, 2019. 2, 18

[Llo96] Seth Lloyd. Universal quantum simulators. Science , 273(5278):1073-1078, 1996. 25

[MBB24] Nimrah Mustafa, Aleksandar Bojchevski, and Rebekka Burkholz. Are gats out of balance? Advances in Neural Information Processing Systems , 36, 2024. 18

[MGL + 23] Gr´ egoire Mialon, Quentin Garrido, Hannah Lawrence, Danyal Rehman, Yann LeCun, and Bobak Kiani. Self-supervised learning with lie symmetries for partial differential equations. Advances in Neural Information Processing Systems , 36:28973-29004, 2023. 25

[MHRB17] Zakaria Mhammedi, Andrew Hellicar, Ashfaqur Rahman, and James Bailey. Efficient orthogonal parametrisation of recurrent neural networks using householder reflections. In International Conference on Machine Learning , pages 2401-2409. PMLR, 2017. 17

[MKB + 20] Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. In ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+ 2020) , 2020. 7, 28, 33

[MLL + 23] Liheng Ma, Chen Lin, Derek Lim, Adriana Romero-Soriano, Puneet K Dokania, Mark Coates, Philip Torr, and Ser-Nam Lim. Graph inductive biases in transformers without message passing. In International Conference on Machine Learning , pages 2332123337. PMLR, 2023. 18, 30

[MMM21] Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Learning with invariances in random features and kernel models. In Mikhail Belkin and Samory Kpotufe, editors, Proceedings of Thirty Fourth Conference on Learning Theory , volume 134 of Proceedings of Machine Learning Research , pages 3351-3418. PMLR, 15-19 Aug 2021. 1

[MMS + 17] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 , 2017. 20

[MP17] Junjie Ma and Li Ping. Orthogonal amp. IEEE Access , 5:2020-2033, 2017. 18

[MPBK24] Sohir Maskey, Raffaele Paolino, Aras Bacho, and Gitta Kutyniok. A fractional graph laplacian approach to oversmoothing. Advances in Neural Information Processing Systems , 36, 2024. 18, 19

[MQ02] Robert I McLachlan and G Reinout W Quispel. Splitting methods. Acta Numerica , 11:341-434, 2002. 25

[NA05] Yasunori Nishimori and Shotaro Akaho. Learning algorithms utilizing quasi-geodesic flows on the stiefel manifold. Neurocomputing , 67:106-135, 2005. 27

[NHN + 23] Khang Nguyen, Nong Minh Hieu, Vinh Duc Nguyen, Nhat Ho, Stanley Osher, and Tan Minh Nguyen. Revisiting over-smoothing and over-squashing using ollivier-ricci curvature. In International Conference on Machine Learning , pages 25956-25979. PMLR, 2023. 28, 32

[NSB + 22] Quynh T Nguyen, Louis Schatzki, Paolo Braccia, Michael Ragone, Patrick J Coles, Frederic Sauvage, Martin Larocca, and Marco Cerezo. Theory for equivariant quantum neural networks. arXiv preprint arXiv:2210.08566 , 2022. 18

[Pet06] Peter Petersen. Riemannian geometry , volume 171. Springer, 2006. 24

[PGC + 17] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017. 30, 33

[PKD + 23] Oleg Platonov, Denis Kuznedelev, Michael Diskin, Artem Babenko, and Liudmila Prokhorenkova. A critical look at the evaluation of gnns under heterophily: Are we really making progress? arXiv preprint arXiv:2302.11640 , 2023. 8, 32, 33

[PMB13] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning , pages 1310-1318. Pmlr, 2013. 2

[PSG18] Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. The emergence of spectral universality in deep networks. In International Conference on Artificial Intelligence and Statistics , pages 1924-1932. PMLR, 2018. 7

[PTPV17] Trang Pham, Truyen Tran, Dinh Phung, and Svetha Venkatesh. Column networks for collective classification. In Proceedings of the AAAI conference on artificial intelligence , volume 31, 2017. 18

[QBY24] Haiquan Qiu, Yatao Bian, and Quanming Yao. Graph unitary message passing. arXiv preprint arXiv:2403.11199 , 2024. 2, 18, 19, 20

[RBM23] T Konstantin Rusch, Michael M Bronstein, and Siddhartha Mishra. A survey on oversmoothing in graph neural networks. arXiv preprint arXiv:2303.10993 , 2023. 1, 6, 18, 28

[RCR + 22] T Konstantin Rusch, Ben Chamberlain, James Rowbottom, Siddhartha Mishra, and Michael Bronstein. Graph-coupled oscillator networks. In International Conference on Machine Learning , pages 18888-18909. PMLR, 2022. 28

[RGD + 22] Ladislav Ramp´ aˇ sek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems , 35:14501-14515, 2022. 18, 28, 30, 33

[S + 77] Jean-Pierre Serre et al. Linear representations of finite groups , volume 42. Springer, 1977. 25, 26

[SBV20] Shlomi, Battaglia, and Vlimant. Graph neural networks in particle physics. Machine Learning: Science and Technology , 2(2):021001, 2020. 1

[SCY + 23] Andrea Skolik, Michele Cattelan, Sheir Yarkoni, Thomas B¨ ack, and Vedran Dunjko. Equivariant quantum circuits for learning on weighted graphs. npj Quantum Information , 9(1):47, 2023. 18

[SF21] Sahil Singla and Soheil Feizi. Skew orthogonal convolutions. arXiv preprint arXiv:2105.11417 , 2021. 2, 5, 17, 18

[SGL18] Hanie Sedghi, Vineet Gupta, and Philip M Long. The singular values of convolutional layers. arXiv preprint arXiv:1805.10408 , 2018. 2, 6, 17, 20

[SHF + 20] Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. arXiv preprint arXiv:2009.03509 , 2020. 18

[SLN + 24] Louis Schatzki, Martin Larocca, Quynh T Nguyen, Frederic Sauvage, and Marco Cerezo. Theoretical guarantees for permutation-equivariant quantum neural networks. npj Quantum Information , 10(1):12, 2024. 18

[SMG13] Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 , 2013. 2, 5, 7

[Str68] Gilbert Strang. On the construction and comparison of difference schemes. SIAM journal on numerical analysis , 5(3):506-517, 1968. 25

[SVV + 23] Hamed Shirzad, Ameya Velingker, Balaji Venkatachalam, Danica J Sutherland, and Ali Kemal Sinop. Exphormer: Sparse transformers for graphs. In International Conference on Machine Learning , pages 31613-31632. PMLR, 2023. 18, 30

[TK21] Asher Trockman and J Zico Kolter. Orthogonalizing convolutional layers with the cayley transform. arXiv preprint arXiv:2104.07167 , 2021. 2, 6, 17, 18, 20, 25, 26, 27, 31

[Tro59] Hale F Trotter. On the product of semi-groups of operators. Proceedings of the American Mathematical Society , 10(4):545-551, 1959. 25

[TRRG23] Jan T¨ onshoff, Martin Ritzert, Eran Rosenbluth, and Martin Grohe. Where did the gap go? reassessing the long-range graph benchmark. arXiv preprint arXiv:2309.00367 , 2023. 18, 30, 31, 36

[TRWG21] Jan T¨ onshoff, Martin Ritzert, Hinrikus Wolf, and Martin Grohe. Walking out of the weisfeiler leman hierarchy: Graph learning beyond message passing. arXiv preprint arXiv:2102.08786 , 2021. 31

[TRWG23] Jan T¨ onshoff, Martin Ritzert, Hinrikus Wolf, and Martin Grohe. Walking out of the weisfeiler leman hierarchy: Graph learning beyond message passing. Transactions on Machine Learning Research , 2023. 18, 30

[VCC + 17] Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017. 18, 28

[WPH + 16] Scott Wisdom, Thomas Powers, John R Hershey, Jonathan Le Roux, and Les Atlas. Full-capacity unitary recurrent neural networks. arXiv preprint arXiv:1611.00035 , 2016. 17, 25

[WSZ + 22] Wu, Sun, Zhang, Xie, and Cui. Graph neural networks in recommender systems: a survey. ACM Computing Surveys , 55(5):1-37, 2022. 1

[WWY20] Rui Wang, Robin Walters, and Rose Yu. Incorporating symmetry into deep dynamics models for improved generalization. arXiv preprint arXiv:2002.03061 , 2020. 1

[WZR + 20] Melanie Weber, Manzil Zaheer, Ankit Singh Rawat, Aditya K Menon, and Sanjiv Kumar. Robust large-margin learning in hyperbolic space. Advances in Neural Information Processing Systems , 33:17863-17873, 2020. 18

[XBSD + 18] Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Pennington. Dynamical isometry and a mean field theory of cnns: How to train 10,000layer vanilla convolutional neural networks. In International Conference on Machine Learning , pages 5393-5402. PMLR, 2018. 7, 17, 18

[XHLJ18] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 , 2018. 18, 30

[XLT + 18] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International conference on machine learning , pages 5453-5462. PMLR, 2018. 18

[YYL20] Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. Advances in Neural Information Processing Systems , 33:17009-17021, 2020. 30, 33

[ZAL18] Zitnik, Agrawal, and Leskovec. Modeling polypharmacy side effects with graph convolutional networks. Bioinformatics , 34(13):i457-i466, 2018. 1

[ZK20] Hao Zhu and Piotr Koniusz. Simple spectral graph convolution. In International conference on learning representations , 2020. 3, 28

Table of Contents

A Extended related works

A.1 Further related literature

Unitary RNNs Unitary neural networks were initially developed to tackle the challenge of vanishing and exploding gradients encountered in recurrent neural networks (RNNs) processing lengthy sequences of data. Aiming for more efficient information learning compared to established nonunitary architectures like the long-short term memory unit (LSTM) [HS97], early approaches ensured unitarity through a sequence of parameterized unitary transformations [ASB16, MHRB17, JSD + 17]. Other methods like the unitary RNN (uRNN) [WPH + 16], the Cayley parameterization (scoRNN) [HWY18], and exponential RNN (expRNN) [LCMR19] parameterized the entire unitary space, maintaining unitarity via Cayley transformations or parameterizing the Lie algebra of the unitary group and performing the exponential map. To eliminate the reliance on the matrix inversion or matrix exponential in these prior algorithms which are computationally expensive in higher dimensions, [KBLL22] showed how to fully parameterize unitary operations more efficiently when applying gradient updates in a low rank subspace.

Groupconvolutional neural networks For convolutional neural networks acting on data in lattices or grids (e.g. images), various algorithms have proposed orthogonal or unitary versions of linear convolutions over the cyclic group [LHA + 19, SF21, TK21, KBLL22]. [XBSD + 18] study the task of initializing convolution layers to be an isometry and thereby are able to train very deep CNNs with thousands of layers without the need for residual connections or batch norm. [SGL18] project convolutions onto an operator-norm ball to ensure the norm of the convolution is within a given range. [LHA + 19] introduced a block convolutional orthogonal parameterization (BCOP) parameterizing a

subspace of orthogonal convolution operations. [SF21] implement orthogonal convolutions by parameterizing the Lie algebra of the orthogonal group and approximating the exponential map. Their approach is a special case of the instances we discuss in our work. [TK21] and [KBLL22] perform convolution in the Fourier domain where convolution is block diagonalized and unitarity can be enforced across each of these blocks. Generally, unitary convolutional architectures do not perform as well as their vanilla counterparts in terms of accuracy on large image datasets such as CIFAR or ImageNet [TK21, SF21, LJW + 19]. Nonetheless, enforcing unitarity here can provide other benefits such as improved robustness to adversarial attacks or added stability with many layers [TK21, KBLL22, SF21, XBSD + 18]. Various architectures for more general geometric domains have been proposed [KT18, CGKW18, CW16, GBH18, WZR + 20, FSIW20], including generalized group convolutional neural networks considered here. To the best of our knowledge, no explicit extensions of unitary convolutions and related stability aspects have been considered in these settings.

GNNs and oversmoothing Our GNN architectures develop on seminal work proposing variants of graph convolution architectures [KW16, BL17]. Graph convolution is one of various types of trainable layers that act on graphs and are equivariant to the permutation group; among those we compare to in our experiments are [KW16, BL17, XHLJ18, TRRG23, HHL + 23, GDBDG23, RGD + 22, MLL + 23, SVV + 23, TRWG23, HYL17, VCC + 17, SHF + 20]. Recently, some work has proposed variants of unitary/orthogonal graph convolution which we discuss in more detail in the next section (App. A.2). Compared to our method, these either only enforce unitarity in part of the transformation or are expensive to compute. For example, [GZH + 22, AEL + 24] propose variants of graph convolution which enforce orthogonality in the feature transformation but not in the message passing update itself. [QBY24] propose a graph unitary message passing algorithm that is in general close to an isometry, but scales poorly with the graph size (see App. A.2). Some recent work has also proposed performing message passing in the complex domain. [MPBK24] show a continuous time message passing update that can be made unitary as discussed in App. A.2. [BSC + 24] propose an equivariant matrix function graph neural network layer that performs a global update by taking a trainable pole expansion of a matrix function of the adjacency matrix. This layer is effective at incorporating global information, but requires matrix inversion operations that can scale poorly with dimension. [MBB24] give a so-called 'balanced orthogonal' initialization for message passing networks using graph attention (GAT). The weight matrix parameters in the GAT layer are initialized as some orthogonal matrix which improves stability for GNNs of the given form with many layers. Nonetheless, the layer itself does not implement an isometry due to the presence of the attention mechanism in the GAT layer.

Various architectures have been proposed to avoid oversmoothing and incorporate nonlocal graph information more generally. This includes approaches that perform small perturbations of the input graph (rewiring [RBM23, FW24]), as well as architectural interventions, such as skip connections [PTPV17, HYL17, XLT + 18], which can improve stability in homophilous settings.

Other settings In the context of Bayesian statistics and graphical algorithms, there are variants of the approximate message passing algorithm which leverage properties of unitarity or orthogonality in the message passing [LGJ + 21, GX15, MP17]. Unitary equivariant operators are also used in the context of quantum computation and quantum variational algorithms [NSB + 22, CNDP + 21, SCY + 23, SLN + 24]. These papers generally study a very different context to that of classical machine learning algorithms. Furthermore, it is unclear whether practical instances of these algorithms offer improvements in comparison to classical machine learning algorithms [AK22, ABKL23]. Separate from quantum machine learning, the unitary graph convolution used here is partly inspired by quantum walks which feature similar unitarity properties [Kem03].

A.2 Comparison with other proposals for unitary message passing

We compare here previous GNN architectures which have proposed variants of complex-valued or isometric message passing. To facilitate this comparison, first let us recall some standard notation and implementations. Given an (possibly normalized) adjacency matrix A ∈ R 𝑛 × 𝑛 acting on 𝑛 nodes and input features X ∈ R 𝑛 × 𝑑 of dimension 𝑑 , a basic linear message passing layer 𝑓 : R 𝑛 × 𝑑 → R 𝑛 × 𝑑 ′ takes the form

where 𝜎 : R → R is a pointwise nonlinearity and W ∈ R 𝑑 × 𝑑 ′ is a parameterized weight matrix transforming the node features. There are variations of the above which normalize the adjacency matrix or add residual connections, but for sake of comparison, we will study the simplified form above. In contrast, the unitary message passing layer we propose takes the form

where X and W are now complex-valued and 𝑡 ∈ R is a parameter controlling the strength of message passing.

[GZH + 22] propose a version of an orthogonal GNN which enforces either exactly or approximately that the matrix W in Eq. (16) is orthogonal, i.e.

They also discuss regularization of node transformations W to bias towards orthogonality. [AEL + 24] also detail a version of a GNN which enforces orthogonality in the matrix W . Wenote that our study focuses on orthogonality in the node evolution during message passing. We discuss in the main text how to combine this with orthogonality in the feature transformation W .

Our work also has overlap with the recently proposed fractional Graph Laplacian approach of [MPBK24]. There, they analyze a continuous-time message passing scheme called the fractional Schr¨ odinger equation taking the form

where 𝛼 > 0 is a chosen parameter and L is the normalized graph Laplacian. Unitarity can be strictly enforced in this continuous time format although [MPBK24] do not take this approach. In fact, solving this continuous-time differential equation for some time 𝑡 , we obtain vec ( X )( 𝑡 ) = exp ( 𝑖𝑡 L 𝛼 ⊗ W ) vec ( X )( 0 ) where vec ( X )( 𝑡 ) is the vectorized version of X at time 𝑡 . Constraining L 𝛼 ⊗ W to be Hermitian obtains a unitary transformation. Beyond enforcing unitarity, our method works over discrete time, and we find it more efficient to parameterize the weight matrix W outside of the exponential map.

A recent preprint proposes a unitary message passing algorithm called graph unitary message passing (GUMP) which sends messages through a larger ring graph constructed from the edges of the original graph [QBY24]. Given a graph 𝐺 with nodes and edges 𝑉 and 𝐸 respectively, the steps of this approach loosely are as follows:

Construct a new directed graph (digraph) 𝐺 ′ with the same nodes and include directed edges ( 𝑖, 𝑗 ) and ( 𝑗 , 𝑖 ) for any undirected edge ( 𝑖, 𝑗 ) in original graph.

Convert 𝐺 ′ to its line graph 𝐿 ( 𝐺 ′ ) : here, each node is an edge in 𝐺 ′ and two nodes in 𝐿 ( 𝐺 ′ ) share an edge if there is a path through the original edges in 𝐺 ′ (e.g. nodes corresponding to edges ( 𝑖, 𝑗 ) and ( 𝑗 , 𝑘 ) ).

Construct new node representations in the line graph where for each edge ( 𝑖, 𝑗 ) we append the node representations [ x 𝑖 , x 𝑗 ] .

Given the adjacency matrix A 𝐿 for 𝐿 ( 𝐺 ′ ) , find permutation matrices P 1 , P 2 which form a block diagonal matrix P ⊺ 1 A 𝐿 P 2 .

Project each block to its closest unitary in Frobenius norm according to the closed form projection operation [Kel75]:

In the above, U( 𝑛 ) denotes the set of unitary matrices in C 𝑛 × 𝑛 .

Invert the permutations obtaining P 1 e A 𝐿 P ⊺ 2 .

Perform GNN convolution using the adjacency matrix constructed previously for 𝑘 layers.

Attach node representations of 𝐿 ( 𝐺 ′ ) to corresponding nodes of 𝐺 and append these to the initial representations x to output the final representations.

As evident above, the approach here is rather meticulous and many details of the approach are left out. We refer the reader to the preprint [QBY24] for those details. Key to this procedure is that unitarity is roughly enforced by constructing the matrix e A 𝐿 which is itself unitary in the vector space of the line graph 𝐿 ( 𝐺 ′ ) . Nonetheless, the map from node representations on the original graph 𝐺 to new node representations on 𝐺 will only be approximately unitary as the additional steps here will not guarantee properties of isometry and invertibility in general.

For sake of comparison, we should note that the complexity of this approach (in number of nodes | 𝑉 | and edges | 𝐸 | ) is at least 𝜔 (| 𝐸 | 2 ) and in practice 𝑂 (| 𝐸 | 3 ) where the most expensive step is the unitary projection which requires a matrix inverse square root. More efficient parameterizations of this procedure or approximations to the projection exist as noted in prior work [KBLL22, LCMR19, LFT20], but this would only improve runtimes to 𝜔 (| 𝐸 | 2 ) at best with additional overhead. Furthermore, the GUMP algorithm expands the memory needed to store hidden states from 𝑂 (| 𝑉 |) to 𝑂 (| 𝐸 |) . In comparison, our approach can be made strictly unitary and is more efficient scaling only a constant factor times standard graph convolution runtime to 𝑂 ( 𝑇 | 𝑉 | | 𝐸 |) where 𝑇 is the truncation in the matrix exponential approximation (see App. C.2) with no change to the way hidden states are stored. Our approach is also noticeably simpler as it only reparameterizes the message passing step to be an exponential map over standard message passing procedures.

B Deferred proofs

First, we review the properties shown in Fact 1 that unitary convolution meets the properties of invertibility, isometry, and equivariance. This fact follows virtually immediately from the definition of unitarity and equivariance.

Proof of Fact 1. The proofs of isometry and invertibility follow directly from the definition of unitarity. What remains to be shown is the equivariance property. For unitary convolution built using Algorithm 1, equivariance can be checked by noting that the exponential map is a composition of equivariant linear operators. For unitary convolution in the Fourier domain (Algorithm 3), equivariance follows from convolution theorems where linear operators appropriately applied in the block diagonal Fourier basis are equivariant [FH13, KT18]. □

As an application, unitary transformations are often used to provide provable robustness guarantees to adversarial perturbations of inputs [TK21, LHA + 19, MMS + 17, SGL18].

Corollary 9 (Certified robustness to adversarial perturbations [TK21]) . A simple consequence of the isometry property of the unitary convolution is that ∥ 𝑓 Uconv ( x )-𝑓 Uconv ( y )∥ = ∥ x -y ∥ so the Lipschitz constant of this function is 1 . Assume neural network classifier 𝑓 : R 𝑛 → R 𝑚 is composed of unitary transformations and 1 -Lipschitz bounded nonlinearities followed by an 𝐿 -Lipschitz transformation and for given input x , it has margin M 𝑓 ( x ) defined as

Then, 𝑓 is certifiably robust (i.e. classification is unchanged) to perturbations x + Δ of magnitude ∥ Δ ∥ < M 𝑓 ( x )( √ 2 𝐿 ) -1 .

In the main text, we noted in Proposition 4 that the above facts in some way cannot be obtained using the standard graph convolution. We restate this here and prove it.

Proposition 4. Let 𝑓 conv : R 𝑛 × 𝑑 → R 𝑛 × 𝑑 be a graph convolution layer of the form

where W 0 , W 1 ∈ R 𝑑 × 𝑑 are parameterized matrices. The linear map 𝑓 (· , A ) : R 𝑛 × 𝑑 → R 𝑛 × 𝑑 is orthogonal for all adjacency matrices A of undirected graphs only if W 1 = 0 and W 0 ∈ 𝑂 ( 𝑑 ) is orthogonal. Furthermore, denoting J A ∈ R 𝑛𝑑 × 𝑛𝑑 as the Jacobian matrix of the map 𝑓 conv (· , A ) , for any choice of W 0 , W 1 , there always exists a normalized adjacency matrix ˆ A such that

where ∥ M ∥ is the operator norm of matrix M .

Proof. Note that for (a potentially normalized) adjacency matrix A , the Jacobian J A in the basis of vec ( X ) is equal to

Thus, for the map to be orthogonal, it must hold that

First, let A 0 be the empty graph on two nodes so A = 0 , then

which means W 0 must be orthogonal if J A 0 is orthogonal. A simple instance with the desired structure can be constructed as follows: Consider the graph on two nodes, which is connected with adjacency matrix A 1 . Here, we have

Note that if W 0 is orthogonal, then W 1 = 0 if J A 1 J ⊤ A 1 = I . This proves the first part of the proposition.

For the second part, we take traces and note that

In contrast, Tr ( I ) = 2 𝑑 . Therefore, for any choice of the values of ∥ W 0 ∥ 2 𝐹 , ∥ W 1 ∥ 2 𝐹 , it must hold that either

W.l.o.g. assume the first event holds. Denoting the singular values of a matrix J A 1 J A 1 -I as 𝑠 1 , . . . , 𝑠 2 𝑑 , we have

Rearranging, we obtain the final result.

□

One consequence of the above fact is that fully parameterized unitary graph convolution requires higher order powers of A to be implemented. This is essentially one reason why the exponential map (or some approximation thereof) is needed.

We now show that the Rayleigh quotient defined in Definition 5 is invariant under unitary transformations. As before, this will follow virtually directly from the unitarity properties. We restate the proposition below.

Proposition 6 (Invariance of Rayleigh quotient) . Given an undirected graph G on 𝑛 nodes with normalized adjacency matrix e A = D -1 / 2 AD -1 / 2 , the Rayleigh quotient 𝑅 G ( X ) = 𝑅 G ( 𝑓 Uconv ( X )) is invariant under normalized unitary or orthogonal graph convolution (see Definitions 1 and 3).

Proof. First, consider separable unitary convolution (Definition 1). The Rayleigh quotient takes the form

The Frobenius norm is invariant under unitary transformations so the denominator ∥ exp ( 𝑖 e A ) XU ∥ 2 𝐹 = ∥ X ∥ 2 𝐹 . For the numerator we have by the cyclic property of trace

or

The matrices exp (-𝑖 e A ) , ( I -e A ) , exp ( 𝑖 e A ) all share the same eigenbasis so they commute and thus

Thus, we have

Similarly for Lie unitary/orthogonal convolution (Definition 3), the isometry property guarantees ∥ X ∥ 2 𝐹 = ∥ 𝑓 Uconv ( X )∥ 2 𝐹 . Furthermore, when viewed as a linear map in the basis of vec ( X ) ∈ C 𝑛𝑑 (i.e. entries of X viewed as a vector), 𝑓 Uconv can be written as a linear map of the form

where W is the feature transformation matrix in Definition 3. Finally, note that exp ( A ⊗ W ⊤ ) commutes with A so

Multiplying the above by ∥ X ∥ -2 𝐹 recovers 𝑅 G ( X ) .

□

In contrast, we gave an example (Proposition 7) where the Rayleigh quotient decays with high probability for vanilla convolution, which we restate below.

Proposition 7. Given a simple undirected graph G on 𝑛 nodes with normalized adjacency matrix e A = D -1 / 2 AD -1 / 2 and node degree bounded by 𝐷 , let X ∈ R 𝑛 × 𝑑 have rows drawn i.i.d. from the uniform distribution on the hypersphere in dimension 𝑑 . Let 𝑓 conv ( X ) = e AXW denote convolution with orthogonal feature transformation matrix W ∈ 𝑂 ( 𝑑 ) . Then, the event below holds with probability 1 -exp (-Ω ( √ 𝑛 )) :

Proof. Note that





By symmetry properties of the uniform distribution Unif ( 𝑆 𝑑 -1 ) on the hypersphere, E [ XX ⊤ ] = I . Thus,

Furthermore, treating 𝑅 G ( X ) as a function of its node features x 1 , . . . , x 𝑛 where x 𝑖 = X ⊤ 𝑖, : , we have by the Azuma-Hoeffding inequality that

The above can be shown by noting that the operator norm of I -e A is bounded by two and changing the values of any x 𝑖 changes 𝑅 G ( X ) by at most 4 / 𝑛 . Therefore, setting 𝜖 = 𝜀𝑛 -1 / 4 , we get that with probability 1 -exp (-Ω ( 𝜀 2 √ 𝑛 )) ,

For 𝑅 G ( 𝑓 conv ( X )) , we have





Again, using the Azuma-Hoeffding argument as before, the numerator above concentrates around its expectation as

A similar statement holds for the denominator ∥ e AX ∥ 2 𝐹 . Furthermore, we have by symmetry properties of the distribution of X and linearity of expectation:

Combining the above facts and applying the union bound, we have that with probability 1 -exp ( Ω ( 𝜀 2 √ 𝑛 )) ,

In the last equality, we use the fact that A has bounded degree and thus Tr ( e A 2 ) = Θ ( 𝑛 ) . □

C Background on representation theory, Lie groups, exponential map, and related approximations

C.1 Matrix Lie groups

We give a brief overview of matrix Lie groups and Lie algebras here and recommend [Hal15] for detailed exposition. Matrix Lie groups are subsets of invertible matrices that form a differentiable manifold formally defined below.

Definition 10 (Matrix Lie groups [Hal15]) . A matrix Lie group is any subgroup of 𝐺𝐿 ( 𝑛, C ) with the property that any convergent sequence of matrices M 𝑚 ∈ C 𝑛 × 𝑛 in the subgroup converge to a matrix M that is either an element of the subgroup or not invertible ( i.e., not in 𝐺𝐿 ( 𝑛, C ) ).

The orthogonal and unitary groups whose definitions are copied below meet these criteria.

Acrucial operation in matrix Lie groups is the exponential map. Given a matrix M : C 𝑛 × 𝑛 which is an endomorphism End ( C 𝑛 ) on the vector space C 𝑛 , the exponential map exp : End ( C 𝑛 ) → End ( C 𝑛 ) returns another matrix (endomorphism) and is defined as

The exponential map, as we will show below, maps from the Lie algebra to the Lie group. It also has an interpretation over Riemannian manifolds which we do not discuss here. We refer the reader to a

textbook [Pet06] or prior work in machine learning [LCMR19] for further details of that connection. For compact groups, the exponential map is a smooth map whose image is the connected component to the identity of the Lie group [KJ08, Hal15]. The Lie algebra is the tangent space of a Lie group at the identity element. To see this, note that

Note that exp ( 0 ) = I . The above gives us the Lie algebra to a given group.

Definition 11 (Lie algebra [Hal15]) . Given a matrix Lie group 𝐺 , the Lie algebra 𝔤 of 𝐺 is the set of matrices X such that 𝑒 𝑡 X ∈ 𝐺 for all 𝑡 ∈ R .

As an example, consider the unitary group where given a matrix U ∈ 𝑈 ( 𝑛 ) and X ∈ 𝔲 ( 𝑛 ) . Here, we have

C.2 Exponential map

First, let us recall the definition of the exponential map. Given a linear operator L : V → V which is an endomorphism End (V) on a vector space V , the exponential map exp : End (V) → End (V) is defined as

Applying the exponential map of a linear operator to a given vector in a vector space of dimension 𝑛 to computer precision can scale in practice as 𝑂 ( 𝑛 3 ) similarly to performing an eigendecomposition. In fact, for matrices M ∈ C 𝑛 × 𝑛 which have an eigendecomposition M = UDU † (for skew-Hermitian matrices M ∈ 𝔲 ( 𝑛 ) , the spectral theorem guarantees the existence of an eigendecomposition.), exp ( M ) = U exp ( D ) U † where exp ( D ) can be easily implemented by performing the exponential elementwise on the diagonal entries of D . However, in practice, we can exploit approximations which avoid expensive operations such as eigendecompositions.

Taylor approximation Often the simplest and most efficient approximation is a 𝑘 -th order truncation to the Taylor series

where conv 𝑝 : V → V indicates the composition of the operator conv 𝑝 times. By Taylor's theorem, one can bound the error in this approximation as

where ∥ conv ∥ denotes the operator norm of conv. Therefore, error decreases exponentially with 𝑘 . In practice, we find setting 𝑘 = 12 is sufficiently accurate for the GNNs we train in our experiments. Of course, 𝑘 can be increased in settings with larger graphs or more training time where instabilities are likely to arise. We also by default perform unitary convolution with normalized adjacency matrices D -1 / 2 AD -1 / 2 ( D is the diagonal degree matrix) so that the operator norm of the linear convolution operation is bounded by one.

and

Pad´ e approximant Pad´ e approximations are optional rational functional approximations of a given function up to a given order. In unitary settings, Pad´ e approximants are often preferred because they return a unitary operator. In contrast, Taylor approximations discussed previously only return a linear operator which can be made arbitrarily close to a unitary operator. Pad´ e approximants of order 𝑘 take the form

where 𝑝 𝑘 , 𝑞 𝑘 are degree 𝑘 polynomials. The degree 𝑘 Pad´ e approximant agrees with the Taylor expansion up to order 2 𝑘 . The first order Pad´ e approximant, also known as the Cayley map, has been used in prior neural network implementations [WPH + 16, HWY18, TK21] and takes the form

One practical drawback of Pad´ e approximants are that they typically require implementations of a matrix inverse or approximations thereof which render this more challenging to implement. In our setting, we found that the Taylor approximation sufficed in accuracy and stability so we did not implement this.

Finally, we should remark that Pad´ e approximants can be used to pre-compute matrix exponentials to computer precision [Hig09, AMH10, AMH09]. [LCMR19] use these techniques to parameterize unitary layers accurately for RNNs. One could pre-compute matrix exponentials of adjacency matrices in our setting to speed up training, though we have not implemented this in our experiments. We leave this for future work.

Other implementations of exponential map We briefly mention here for sake of completeness a different approach to approximating the exponential map based on the Baker-Campbell-Hausdorff formula [Hal15] which states that for matrices X , Y ∈ C 𝑛 × 𝑛 :

The Lie-Trotter approximation uses the above fact to implement the matrix exponential of a sum of matrices as a product of matrix exponentials over elements of the sum. This method is often used in high dimensional spaces where the output of the exponential map on each Lie algebra generator is known. The first order expansion takes the form [Tro59, CST + 21]

where 𝑘 is a positive integer controlling the level of approximation. The above is accurate to order 𝑂 ( 1 / 𝑘 ) , and higher order accurate schemes exist. These methods are commonly used in machine learning, quantum computation, and numerical methods 10 [CST + 21, MGL + 23, Llo96, Str68, MQ02]. One advantage of this approach is that the approximation applied to an operator in the Lie algebra always returns an element in the Lie group since the approximation is a product of Lie group elements themselves.

D Fourier implementation of group convolution

In the main text, we described a generalized procedure to implement unitary convolution with parameterized operators in the Lie algebra of a group. We complement that with a mostly informal discussion on how to implement unitary convolution in the Fourier domain. The algorithms are summarized in Algorithm 3 and Algorithm 2.

Before detailing the algorithm in Algorithm 3, we provide a brief overview of Fourier-based convolutions over arbitrary groups, which generalizes the classical Fourier transform to that over arbitrary groups. We recommend [S + 77] for a more complete and rigorous background. Throughout this, we will assume that groups are finite, though this can be generalized to other groups, especially those that are compact.

Algorithm 2 Unitary map from Lie algebra

Input: equivariant linear operator L ∈ C 𝑛 → C 𝑛

Input: vector x ∈ C 𝑛

1: e L = 1 2 ( L - L † ) (skew symmetrize operator)

2: return exp ( e L )( x ) (or approximation thereof)

Arepresentation of a group is a map 𝜌 : 𝐺 → F 𝑑 × 𝑑 for a given field F such that 𝜌 ( 𝑔 ) 𝜌 ( 𝑔 ′ ) = 𝜌 ( 𝑔𝑔 ′ ) (homomorphism). A representation is reducible if there exists an invertible matrix 𝑄 ∈ F 𝑑 × 𝑑 such that 𝑄𝜌 ( 𝑔 ) 𝑄 -1 = ⊕ 𝑘 𝑖 = 1 𝜌 ′ 𝑖 ( 𝑔 ) is a direct sum of at least 𝑘 ≥ 2 representations 𝜌 ′ 1 , . . . , 𝜌 ′ 𝑘 . A representation is irreducible if such a decomposition does not exist. For any finite group 𝐺 , a system of unique irreps 𝜌 1 , . . . , 𝜌 𝐾 always exists. It holds that ˝ 𝑖 dim ( 𝜌 𝑖 ) 2 = | 𝐺 | for any such set of unique irreducible representations [S + 77]. Here, the uniqueness is to eliminate redundancies due to the fact that any irrep 𝜌 can be mapped to a new one by an invertible transformation 𝜌 ′ ( 𝑔 ) = 𝑄𝜌 ( 𝑔 ) 𝑄 -1 .

The Fourier transform of a function 𝑓 : 𝐺 → C with respect to a set of irreducible representations (irreps) { 𝜌 𝑖 } 𝑚 𝑖 = 1 is given by:

For abelian groups these irreps are all one dimensional. The set of 𝜌 𝑖 ( 𝑢 ) for the cyclic group for example correspond to the entries of the discrete Fourier transform matrix. For non-abelian groups, there always exists an irrep which is at least of dimension 2.

In Algorithm 3, we denote the Fourier transform F : C 𝑛 → ⊕ 𝑚 𝑖 = 1 C 𝑑 𝑖 × 𝑑 𝑖 as a map that takes in the inputs to a function 𝑓 and outputs a direct sum of the Fourier basis of the function ˆ 𝑓 over the irreps. Given two functions 𝑓 , 𝑔 : 𝐺 → C , their convolution outputs another function ( 𝑓 ∗ 𝑔 ) : 𝐺 → C and is equal to

In the main text, we describe these operations as vector operations over a vector space C 𝑛 . Setting 𝑛 = | 𝐺 | and taking the so-called regular representation as maps acting on C | 𝐺 | can recover the form in the main text. Finally, the Fourier transform of their convolution is the matrix product of their respective Fourier transforms:

where 𝜌 𝑖 𝑚 𝑖 = 1 are the irreps of 𝐺 . The above assumes that all the functions have input domain over the group. This is not strictly necessary and generalizations exist which map functions on homogeneous spaces to the setting above [KT18].

The general implementation of Fourier convolution is given in Algorithm 3. Here, one employs a Fourier operator which block diagonalizes the input into its irreducible representations or some spectral representation. Then, applying blocks of unitary matrices in this representation and inverting the Fourier transform implements a unitary convolution. The details will depend on the particular form of the Fourier transform and irreducible representations. This method is often preferred when filters are densely supported and efficient implementations of the Fourier transform are obtained. Previous implementations have been designed for CNNs [TK21, KBLL22].

Example 3 (Convolution on regular representation (Fourier basis)) . Continuing the previous example, assume the group 𝐺 has unitary irreducible representations 𝜌 1 , . . . , 𝜌 𝑚 (irreps) where 𝜌 𝑖 : 𝐺 → C 𝑑 𝑖 × 𝑑 𝑖 . The group Fourier transform maps input function 𝑥 : 𝐺 → C to its irrep basis as [FH13, KT18]

and group convolution is now block diagonal in this basis

Implementing unitary convolution requires that ˆ 𝑚 ( 𝜌 ) is unitary for all irreps 𝜌 .

E Architectural considerations

Handling complex numbers and enforcing isometry in neural networks requires changes to some standard practice in training neural networks. We summarize some of the important considerations here and refer the reader to surveys and prior works for further details [BQL21, TK21, LCMR19, KBLL22].

Handling different input and output dimensions Unitary and orthogonal transformations are defined on input and output spaces of the same dimension. Maintaining isometry for different input and output dimensions formally requires manipulations of the Stiefel manifold as studied in various prior works [LCMR19, LFT20, NA05, JD15]. When the input dimension is less than that of the output dimension, one simple way to implement semi-unitary or semi-orthogonal convolutions via standard unitary layers is simply to pad the inputs with zeros to match the output dimensionality. We also often will simply use a standard (unconstrained) linear transformation to first embed inputs in the given dimension that is later used for unitary transformations.

Nonlinearities For handling complex numbers, one must typically redefine nonlinearities to handle complex inputs. We find that applying standard nonlinearities separately to the real and imaginary parts works well in practice in line with other works [BQL21]. To enforce isometry as well in the nonlinearity, we use the GroupSort : R 2 → R 2 activation [TK21, ALG19], which acts on a pair of numbers as

and is clearly norm-preserving. To apply this nonlinearity to a given layer, we split the channels or feature dimension into two separate parts and apply the nonlinearity across the split.

Initialization Given the constraints on unitary matrices and skew-Hermitian matrices, prior work has proposed various forms of initialization that meet the constraints of these matrices. One strategy that has been effective in prior work and proposed in [HSL16, HWY18] is to initialize in 2 × 2 blocks along the diagonal. For skew symmetric matrices, one way of achieving this is to initialize 2 × 2 blocks as

where 𝑠 ∼ Unif (-𝜋, 𝜋 ) for example [HSL16].

Directed graphs Directed graphs present a challenge because their adjacency matrix is not guaranteed to be symmetric. Given an adjacency matrix A ∈ R 𝑛 × 𝑛 of a directed graph, one simple way to proceed is to split the directed graph into its symmetric and non-symmetric parts A = A sym + A nonsym . Here, we assume that for matrix entry A 𝑖 𝑗 , either A 𝑖 𝑗 = A 𝑗𝑖 or A 𝑖 𝑗 A 𝑗𝑖 = 0 (i.e. one of the transposed entries is zero). Then, we set

and

 Finally, one can then perform graph convolution with the skew-Hermitian matrix

We do not work with directed graphs in our experiments and have not implemented this in our code.

F Additional experiments

F.1 Additional results on toy model of graph distance

Fig. 3 shows additional results for the toy model considered in the main text. As a reminder, this task is to learn the graph distance between pairs of randomly selected nodes in a ring graph of 100

0

5

10

test MAE

layers = 5

0

5

10

test MAE

layers = 10

0

50

100

150

200

250

0

5

10

test MAE

layers = 20

model

Residual GCN

GraphCON GCN

Unitary GCN

Vanilla GCN

Spectral Convolution

Graph Attention Network

GPS

nodes. Message passing architectures need at least 50 sequential messages to fully learn this task. As layers are added to the unitary GCN, the network is able to learn the task better. This is in contrast to other message passing architectures which perform worse with additional layers. Apart from message passing architectures, strong performance is also achieved by transformer architectures like GPS with global attention. The networks we study are the vanilla GCN (Residual GCN includes skip connections) [KW16], graph attention network [VCC + 17], spectral convolution [ZK20], transformerbased GPS [RGD + 22], and graph-coupled oscillator network (GraphCON), which is a discretization of a second-order ODE on the graph [RCR + 22]. Hyperparameters and additional network details are reported in App. G.

F.2 TU Datasets

Weperform experiments on the ENZYMES, IMDB-BINARY, MUTAG, and PROTEINS tasks from the TU Dataset database [MKB + 20]. As can be seen in table 3, with the exception of IMDB, a GCN with UniConv layers outperforms all message-passing GNNs tested against on all datasets, by margins of up to 18 percent. We follow the training procedure in [NHN + 23] and report results for GCN and GIN from [NHN + 23]. For GAT and unitary GCN, we tune the dropout and learning rate as detailed in App. G. All results are accumulated over 100 random trials. In addition, in Fig. 4, we show that the unitary GCN maintains its performance over large network depths in contrast to other message passing layers. The deteriation in performance of conventional message passing layers is a likely a consequence of over-smoothing which we show does not occur in unitary graph convolution [CW20, RBM23].

F.3 Dihedral group distance

The dihedral group 𝐷 𝑛 is a group of order 2 𝑛 describing the symmetries (rotations and reflections) of a regular polygon. Its elements are generated by a rotation generator 𝑟 and reflection generator 𝑠 :

In this task, analaogous to the graph distance task in Sec. 5, the goal is to learn the distance between two group elements 𝑔, 𝑔 ′ in the dihedral group 𝐷 𝑛 . Formally, we aim to learn a target function 𝑓 : R | 𝐷 𝑛 | → Z mapping inputs of dimension 2 𝑛 (the vector space of the regular representation) to

0.85

0.80

0.75

U-GCN

0.70

GCN

GIN

0.65

GAT

0.60

0.55

0.50

0

20

40

60

80

100

120

Number of Layers

positive integers. The input space R | 𝐷 𝑛 | is indexed by group elements and convolution is performed on the regular representation of the group. Each data pair ( x 𝑖 , 𝑦 𝑖 ) 𝑖 is drawn as follows:

Draw two group elements 𝑔, 𝑔 ′ ∼ Unif ( 𝐷 𝑛 ) from the uniform distribution over the group 𝐷 𝑛 without replacement.

Set x 𝑖 = e 𝑔 + e 𝑔 ′ where e 𝑔 ∈ R | 𝐷 𝑛 | is a unit vector whose entries are all zero except for the entry corresponding to operation 𝑔 set to one.

Set 𝑦 𝑖 = 𝑑 𝐷 𝑛 ( 𝑔, 𝑔 ′ ) ∈ Z where 𝑑 𝐷 𝑛 ( 𝑔, 𝑔 ′ ) indicates the number of applications of the generators that need to be applied to go from 𝑔 to 𝑔 ′ , i.e.

Fig. 5 plots the performance of three different networks in this task. The vanilla network performs standard convolution in each layer where the group convolution is parameterized over elements of the generator. The residual network is the same architecture but includes skip connections after convolutions. The unitary network applies the exponential map to a skew-Hermitian version of the group convolution as outlined earlier. The unitary network is able to learn this task in fewer layers and with added stability. No hyperparameter tuning was performed in these experiments. Adam optimizer parameters were set to the default setting. All networks have a global average pooling after the last convolution layer followed by a 2 layer MLP to output a scalar. The number of channels is set to 32 for each convolution layer.

F.4 Orthogonal Convolution

To compare orthogonal (real-valued) and unitary convolutional layers, we repeat our experiments on Peptides-func and Peptides-struct from the main text. We report the results in table 4. Edge features

0

200

400

0

20

40

60

value

layers = 20

0

200

400

layers = 50

network

unitary

residual

vanilla

metric

train

test

are not included in this ablation as no edge feature aggregator was included in the architecture. GCNwith OrthoConv layers achieves performance levels similar to UniConv without using complex numbers in its weights.

G Experimental details

For LRGB datasets, we evaluate our GNNs using the GraphGym platform [YYL20]. In Table 1, we list reported results where available from various architectures [KW16, BL17, XHLJ18, TRRG23, HHL + 23, GDBDG23, RGD + 22, MLL + 23, SVV + 23, TRWG23]. Reported results are taken from existing papers as of May 1, 2024. Experiments were run on Pytorch [PGC + 17] and specifically the Pytorch Geometric package for training GNNs [FL19].

Our implementation of unitary graph convolution does not take into account edge features. Thus, for datasets with edge features, at times, we included a single initial convolution layer using the GINE [XHLJ18] or Gated GCN [BL17] architecture which aggregates the edge features into the node features. For instances where this is done, we indicate the type of layer as an 'edge aggregator' hyperparameter in the tables below.

All our experiments were trained on a single GPU (we used either Nvidia Tesla V100 or Nvidia RTX A6000 GPUs). Training time varied depending on the dataset. Peptides datasets trained in no more than 15 seconds per epoch. PascalVOC-SP took about a minute per epoch to train and COCO-SP took about 15 minutes per epoch to train. The PascalVOC-SP and two Peptides datasets are no more than 1 GB in size. The COCO dataset was much larger, about 12 GB in size. The TU and Heterophilous Node Classification datasets are all less than 1GB in size and take only a few second to train per epoch.

Remark 12 (Parameter counting) . LRGB datasets require neural networks to be within a budget of 500k parameters. For fair comparison, complex numbers are counted as two parameters each. Furthermore, to handle constraints for parameterized matrices in the Lie Algebra, we treat the

number of parameters as the dimension of the Lie Algebra (i.e. the number of parameters to fully parameterize the Lie algebra).

Toy model: graph distance All networks consist of the stated number of convolution or transformer layers with feature dimension 128 followed by a global average pooling over nodes. Pooled features are then passed through a single hidden layer MLP with 128 dimension width. For networks with 5, 10 and 20 layers respectively, we set the learning rate to 0 . 0007, 0 . 0003 and 0 . 0001 respectively. Networks are trained with the mean average error (L1) loss. Activation functions for all networks are set to GELU apart from the unitary networks where we choose the norm-preserving activation GroupSort [TK21, ALG19] (see App. E). To ensure unitary layers are also truly normpreserving, we do not include a trainable bias in these layers.

Peptides For the Peptides experiments, the training procedure follows that of [TRRG23]. Unless otherwise stated, we use the Adam optimizer with learning rate set to 0 . 001 and a cosine learning rate scheduler [KB14]. Most hyperparameters were taken from [TRWG21]. For our purposes, we tuned dropout rates in the set { 0 . 1 , 0 . 15 , 0 . 2 } and tested networks of layers in the set { 6 , 8 , 10 , 12 , 14 } selecting the hidden width to fit within the parameter budget. For larger levels of dropout, we found that we needed to train the network for more epochs to achieve convergence. Final hyperparameters are listed in Table 6.

COCO-SP and PascalVOC-SP For the COCO-SP and PascalVOC-SP experiments, our training procedure again follows that of [TRRG23]. We use the Adam optimizer with learning rate set to 0 . 001 and a cosine learning rate scheduler [KB14], unless otherwise stated. For our purposes, we

tuned dropout rates in the set { 0 . 1 , 0 . 15 , 0 . 2 } and tested networks of layers in the set { 6 , 8 , 10 , 12 , 14 } selecting the hidden width to fit within the parameter budget. For larger levels of dropout, we found that we needed to train the network for more epochs to achieve convergence. Final hyperparameters are listed in Table 6.

TUDatasets Our experiments are designed as follows: For a given GNN model, we train on a part of the dataset and evaluate performance on a withheld test set using a train/val/test split of 50/25/25 percent. For all models considered, we record the test set accuracy of the settings with the highest validation accuracy. As there is a certain stochasticity involved, especially when training GNNs, we accumulate experimental results across 100 random trials. We report the mean test accuracy, along with the 95% confidence interval. For the TU Datasets, our training procedure and hyperparameter tuning procedure follows that described in [NHN + 23]. Namely, we fix the depth and width of the network as in [NHN + 23] and tune the learning rate and dropout parameters. Final hyperparameters are listed in Table 7. To ensure fairness and comparability, we conducted the same hyperparameter searches with the baseline models we compare against, but found no improvements over the numbers reported in previous papers. We therefore report their (higher) numbers instead of ours for these models.

Heterophilous Graph Datasets For a given GNN model, we train on a part of the dataset and evaluate performance on a withheld test set using a train/val/test split of 50/25/25 percent, in accordance with [PKD + 23]. Note that we do not separate ego- and neighbor-embeddings, and hence also do not report accuracies for models from the original paper that used this pre-processing (e.g. GAT-sep and GT-sep). Our training procedure generally follows that described in the original paper [PKD + 23]. We use the AdamW optimizer, stop training after no improvement in 200 steps, and including residual connections in the intermediate network layers. For the unitary GNNs, we tuned values for the dropout in { 0 . 2 , 0 . 5 } , number of layers in { 4 , 6 , 8 } and learning rate in { 0 . 001 , 0 . 0001 } . To output a single number, we use a single convolution layer (SAGE convolution) to map from the higher dimensional space to a single number for each node. [PKD + 23] differs in that they have an MLP in between layers; we opt for a more simple approach. Final hyperparameters are listed in Table 8. To ensure fairness and comparability, we conducted the same hyperparameter searches with SAGE and GCN, but generally found no significant differences with respect to the numbers reported in previous papers. We therefore report their usually higher numbers instead of ours for these models.

G.1 Licenses

We list below the licenses of code and datasets that we use in our experiments.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In this work, we support formal claims with proofs and include results for various different datasets which show that the unitary layer we propose enhances stability and achieves strong performance.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have described limitations of our work at various points. Incorporating unitarity is not a panacea and we mention the challenges in dealing with this (e.g. see discussion section).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

Theauthors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All formal statements are proven either in the main text or Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We follow established training procedures as stated in the main text. Code is shared to replicate various experiments. Training details and hyperparameters are reported in App. G.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code is shared in the anonymized zip file to replicate the LRGB Peptides experiments and toy model experiments (both the ring dataset and the dihedral group convolution in App. F). Instructions to initialize our unitary/orthogonal convolution layer and perform experiments are given therein. To handle different datasets, we had to format code differently for the various code bases, and we did not have time to make the code consistent for other graph experiments. Nonetheless, we plan to make all code available before eventual publication and/or during rebuttals if desired. In addition to the submitted code, experimental details are provided in the main text as well as in App. F and G.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We report how we trained different models in the main text and App. G. For LRGB experiments, we report in comparison to reported results and follow the training procedure in [TRRG23] as mentioned in the main text. Hyperparameters and other training details are reported in App. G for all experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars included in all experiments and tables. Details are provided in App. G.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: A single GPU is needed to run an individual experiment; details are provided in App. G.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work presents foundational research on improving learning on graphs and groups. Though this work has impact on the research community in machine learning, wedonot see any clear links to questions of privacy, misuse, fairness, or other major societal impacts here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work is largely foundational and theoretical. All models and datasets are relatively small and scientific in nature.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Datasets and models are cited throughout and licenses are listed in App. G.1. Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The unitary convolution layers are shared in the attached code and documented in the text. Apart from the simple toy datasets of graph distance on a ring or group, no new datasets are introduced here.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces two unitary graph convolution operators (UniConv and Lie UniConv), and studies their performance and ability to avoid over-smoothing even in deep Graph Neural Networks. UniConv (short for separable unitary convolution) takes the form f_{Uconv} = exp(iAt)XU (with A the adjacency matrix, i the imaginary unity, and U a unitary operator), while Lie UniConv (short for Lie Unitary Convolution) is defined as f_{Uconv} = exp(g_conv)(X) (with g_conv(X)=AXW, and W a skew-symmetric matrix). In both cases, the exponential map is approximated with a truncated Taylor series to contain complexity and avoid any form of eigen-decompositon. In experimental evaluation, GNNs implemented with UniConv achieved good performance on a variety of real-world datasets, while Lie UniConv was the only model to succeed in a toy example exhibiting long-range dependencies when compared to classic GNNs (such as GCN, GAT and Residual GCN).\n\nSTRENGTHS:\nThe paper is generally well written and with a good introduction exhibiting the oversmoothing effect that one can observe with a classic GCN approach, and how the proposed unitary convolution can avoid that. In terms of the presentation of the methods, I didn’t have particular difficulties following the general idea of the authors, although there are some items I need to clarify with them (please see below in the weaknesses of the paper). Experiments appear overall good (though with some limitations on the Lie Uniconv method), with UniConv achieving good performance on a variety of benchmarks (including datasets showing long range dependencies and heterophilic graphs).\n\nWEAKNESSES:\nWhile I generally appreciated the paper (hence my slightly positive score), I’m not fully convinced by the Lie UniConv approach. The authors propose the two methods I highlighted above as a solution to avoid overfitting. As overfitting is caused by a contraction of all frequencies associated to all but the lowest frequency of the Laplace operator, converting the adjacency matrix of the provided graph into a unitary operator (as it is done in UniConv) makes sense to avoid the issue (as all eigenvalues of the obtained diffusion operator now have norm equal to 1). However, the Lie UniConv approach doesn’t go in that direction, but rather applies an exponential map on operator g_conv to obtain an orthogonal map. In this case, why the exponential of such operator is an orthogonal map is not specified in the paper, and I struggle to understand the rationale behind this approach. In this direction, in the experimental evaluation of the paper, I didn’t see any result using Lie Uniconv besides the experiment of the synthetic long rage benchmark I highlighted above. On top to this, on such benchmark, UniConv is not applied in the paper. If this was achieving comparable, or even superior, results to Lie Uniconv, then there would not be any result in the paper showing the benefit of using the Lie Uniconv method.\n\nI would kindly ask the authors if they could clarify my doubts above, and present additional results with Lie Uniconv in the rebuttal (unless I missed them somehow in the paper).\n\nIn addition to this, I would like to point out that the results presented in Table 2 are incomplete, and while the proposed Uniconv outperforms the reported methods, it is possible to see better performing solutions on Roman E in “A critical look at the evaluation of gnns under heterophily: Are we really making progress?”. I would thus ask the authors to report all the relevant results from the paper, to ensure a fair comparison.\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to use so-called \"unitary group convolutions\" in graph neural networks, with main motivation to do steps to overcome gradient collapse or explosion and oversmoothing effects in graph neural networks under equivariance constraints. This fits within a more general line of work in which unitary constraints are applied for regularizing various learning architectures.\n\nSTRENGTHS:\nThe use of unitary constraints is a sound idea for use in GNNs, in principle. \nEdit after rebuttal: It is also a novel idea, different than usual normalizations and ensuring higher nondegeneracy.\n\nThe explanations are clear and the paper is mostly well written (see \"questions\" part for some possible doubts).\n\nWEAKNESSES:\nEdit: after rebuttal, the main issue of originality is 100% solved, and the points in questions 3 and 5 are fully solved. The authors have proposed better experiments. This makes my concern in high part mute, the only small (and out of the scope of this work) point is that it will be great to see more verifications of oversmoothing control of beyond the (already convincing) Rayleigh Quotients preservation, which is proved in the work. As said above, this is for future work, and requires much more space, so the paper is very strong as is.\n---------------------------\n\nThe originality of this approach is a bit limited, since normalizing message passing is not a very novel idea. \nDoing it with unitary constraints has not been explored before for GNNs. It has been explored in other settings though.\n\nThe mathematical setup is sketchy at times, it needs better care. In particular, it is not clear if the proposed convolution is really a unitary operation (see question 3 and the crucial question 5 in \"questions\" section, especially).\n\nThe experiments show that normalizing is a good idea, but they don't fully convince that unitary constraints is the best way to normalize, since comparisons are done mainly against non-normalized GNNs. One has the impression that this is mainly a first exploration of a theme, and it would need more careful benchmarking, since the computational cost for implementing this particular normalization is not always negligible.\n\nQUESTIONS:\nWhat's the relative benefit compared to more naive normalizations? \n\n\nMinor correctionsl/doubts:\n1) lines 29-31: \"it has been widely observed that group-convolutional networks suffer from instabilities as their depth increases\" -- if it's widely observed, please indicate a few references where this was observed?\n\n2) line 103: \"linear convolution\" -- as opposed to what other kind of convolution? isn't convolution always linear?\n\n3) line 144: the following is an important point/issue. \n\"When W contains only real-valued entries, the above returns an orthogonal map\" -- I don't follow this, why is that? this statement would require a proof, because it far from obvious, and in general it is false. Recall that a product of matrix exponentials is in general not the exponential of the product, so $exp (g_{\\mathrm{conv}})(X) \\neq e^A X e^{W^*}$ in this case.\n\n4) I don't see what's the importance / relevance of examples 1 and 2 from Section 3. I think these are standard examples, they can be skipped or moved to the appendix. \n\n5) lines 217-218: this probably the most important theory issue of the paper.\n \"the Rayleigh quotient is invariant to unitary transformation giving a proof that unitary graph convolution avoids oversmoothing\" \nOne can vaguely agree that Rayleigh quotient collapse may in some way be correlated to oversmoothing, but saying that Prop 6 \"gives a proof that unitary convolution avoids oversmoothing\" is unjustified without further proofs/details. In particular some form of a result saying that \"Rayleigh quotients do not collapse implies there is no oversmoothing\" is missing. Without this, the main claim of the paper hinges on \"vague intuition\" about RQ's and the paper is much less valuable at a theory level.\n\n6) Definition 6 (dynamical isometry): what iis the connection of this definition with the definitions from section 3.1? This part on dynamical isometry is at present very weakly (if at all) connected to the core of the paper, and I did not understand how the authors are linking it to the main definitions. See question 3 related to this.\n\n7) The experiments show performance on some benchmarks, but I would have been more convinced if the unitary convolution proposed method would have been compared to other simpler normalization methods which have similar effects on oversmoothing and gradient vanishing/explosion.\n\nLIMITATIONS:\nMain limitations are already included in the \"questions\" section, esp questions 3,5,7.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper the authors propose a mathematically consistent treatment of unitary convolutions in the context of neural networks defined on graphs and groups. In the graph neural network context, the main idea is to transform a standard linear convolutional layer $f_\\text{conv} = \\mathbf{AXW}$ with $\\mathbf{A}$ the adjacency matrix, $\\mathbf{X}$ the signal input matrix and $\\mathbf{W}$ the parameterize into a linear layer $f_\\text{Uconv}$ that is unitary (norm preserving), taking into account both the matrices $\\mathbf{A}$ and $\\mathbf{W}$ (previous literature focus on $\\mathbf{W}$ or perform this in a less natural way (with worse computational complexity) [QBY24]. The main idea revolves around the following two steps: (1) project the operation into matrix Lie algebra (e.g., the space of skew-symmetric matrices) (2) apply the exponential map on the projected operation to obtain an unitary operation. In the graph setting the authors do this in two ways: the first is called \"Separable unitary graph convolution (UniConv)\" and takes the form $f_{\\text{Uconv}} (\\textbf{X}) = \\exp(i\\textbf{A}𝑡)\\textbf{X}\\textbf{U}, \\textbf{U} \\textbf{U}^{\\dagger} = \\textbf{I}$. The second is called \"Lie orthogonal/unitary graph convolution (Lie UniConv)\" and is defined by applying an exponential map on a convolution with skew-symmetric weight matrices in Lie algebra, having the advantage of being parameterizable over the reals only. The authors then discuss how to approximate the exponential map (they resort to simple Taylor truncation).  Subsequently, the authors develop the theory in the case of general groups and show to ways of obtaining unitary convolutions: the first case generalizes the two methods defined for graphs, by first skew-symmetrizing the equivariant map and then applying exp; the second is defined as a (block-diaognal) convolution in the Fourier domain followed by inverse transform. Following the authors show basic properties of the proposed method (invertibility, isometry and equivariance) and prove favorable results which mainly follow by isometry (avoiding oversmoothing, avoiding vanishing/exploding gradients). In the experimental section, the authors perform experiments on toy ring graph (on Lie UniConv) and the standard benchmarks LRGB and Heterophilous Graph Dataset (on UniConv) (also showing group convolution in Appendix on Dihedral group $D_n$).\n\nSTRENGTHS:\n- Well founded mathematical formalism, defining unitarity in message passing graph neural networks in a natural way.\n- Strong performance over other message passing algorithms, comparable performance with respect to graph attention based models. On very large graphs attention could be prohibitive so UConv could perform well in that situation.\n- The paper has an extensive and complete literature overview, comparing with related literature in very detailed manner (the Appendix on related work relative to unitary convolutions is a very good addition and clearly let us understand the contribution).\n- Theoretical analysis on oversmoothing and vanishing/exploding gradients shows the advantage on adopting UConvs in a quantitative way.\n\nWEAKNESSES:\n- There aren't many experiments on Lie UniConv. From what I understand, the only experiment with this layer is the graph ring experiment in Figure 2. Also it would have been interesting to see comparison between the two type of convolutions (UniConv vs Lie UniConv), with the more realistic datasets, in order to asses which one performs better in practice.\n\n- The part on unitary maps in Fourier basis feels somehow detached from the general discourse of the paper. I believed the author added it for completeness but (1) the theory to understand it is relatively advanced (had to study externally some representation theory to understand well the definition of Fourier transform on finite groups); a short primer on representation theory in the Appendix, similar to the one for Lie groups / algebras, would have been of great help. (2) There are no experiments in this setting. (3) some parts are a little bit shaky, for example in: ```One can also generally implement convolutions in the (block diagonal) Fourier basis of the graph\nor group (Algorithm 2). Here, one employs a Fourier operator which block diagonalizes the input into its irreducible representations or some spectral representation.```, if one considers using the eigendecomposition of some graph Laplacian as a spectral basis, the (graph) Fourier transform is not given by a set of matrices varying over irreps but a signal of the same dimensionality of the input function. I do not think one can use the same approach there.\n\n- Sometimes authors leave facts implicit and the writing could be improved in key places. For example I had difficulty understanding why Lie UniConv returns an unitary map before looking at the tensor product version of $g_{\\text{conv}}$ (Eq. (36), Proposition 6). Also the statements in Propositions 4 and 7 are not really clear from a first reading, some more intuition about the statements should be given.\n\nQUESTIONS:\n- I don't see how the graph convolutions defined are particular cases of the group convolutions described. Is it the symmetric group S(n) acting on the graph?\n- Line 122: Why UniConv has a \"tensor product nature\"? From what I understand the tensor product is more related to the Lie UniConv because you can write it as $\\exp(\\mathbf{A} \\otimes \\mathbf{W}^T)$.\n- Line 186: \"We set input/output representations to be equal so that the exponential map of an equivariant operator is\nitself equivariant\": Assuming this does not limit the class of groups or transforms we can use? Not all groups have irreps of same dimensions. Or we say that the transforms are equivariant only in this sense? Also why exponential map is equivariant only if the input output representations are equal?\n- Line 212: is (I − A) a type of graph Laplacian?\n- Line 272: Why authors don't compare to new architectures as well on this task, like in Table 1? Also I see in Appendix that GPS it is performing pretty well w.r.t. Unitary GCN but it was not put in this graph (maybe to show better improvements?).\n- Line 763: I think authors should say that in case of Algo 2 this follows because Fourier transform is unitary. It is not clear if and why this should be the case in finite group setting.\n- Line 766: \"For unitary convolution in the Fourier domain (Algorithm 2), equivariance follows from convolution theorems where linear operators appropriately applied in the block diagonal Fourier basis are equivariant\". This is too succinct and should be more expanded formally.\n- Line 821: missing trace on following lines and last line should not be put there ($= R_{\\mathcal{G}(\\mathbf{X})}$)\n\nLIMITATIONS:\nThe authors have discussed limitations of their contribution adeguatly.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces convolutional layers for GNNs that are based on group-convolutions. Some of the proposed layers have interesting theoretical properties such as avoiding oversmoothing or vanishing gradients. Experimentally, these layers are able to aggregate information across long distances and achieve strong results on real-world benchmarks.\n\nSTRENGTHS:\n- **(S1 - Novelty)** The contributions in this paper seem novel.\n\n- **(S2 - Significance)** From a theory perspective, some of the proposed operations have interesting theoretical properties such as that they can avoid oversmoothing or vanishing gradients.\n\n- **(S3 - Significance)** The authors perform extensive experiments on real-world datasets and achieve strong results on most of them.\n\nWEAKNESSES:\n- **(W1 - Clarity and Significance)** The reader is not guided through the paper and it is difficult to understand the flow of ideas. I will give some examples of this:\n\n   - In Section 3.1 (Lie) UniConv layers are introduced which is the main contribution of the pape. However, to me it was not clear that this is the central definition. \n\n   - In the following Section 3.2, the paper generalizes the setting of unitary convolutions and gives several examples. Yet, to me it is unclear what this section intends to communicate and how it is relevant to the rest of the paper. \n\n   - Another example of this is the “Vanishing/Exploding gradients”-part of Section 4, here dynamical isometry is defined as a measure as a property to analyze exploding gradients and an example is given that a combination UniConv together with a yet completely undefined activation function (GroupSort) is “perfectly dynamically isometric”. This raises many questions to the reader: Why is an example important enough to warrant the definition of Dynamical isometry? How does this relate to the main contribution of the paper? Are vanishing / exploding gradients a problem with GNNs? What is GroupSort and is it actually used in practice? \n\n- **(W2 - Clarity)** The experiment section is difficult to read and does not properly explain the experiments or the design decisions. For example, for the graph distance dataset the model is not mentioned in the text. In this case, the model is GCN with the novel Lie UniConv layer (Figure 2). However, it is not explained why the Lie UniConv layers are used and not the UniConv Layers. Similarly, for the experiments on LRGB UniConv layers are used but it is not explained why Lie UniConv layers are not tried.\n\n**To sum up.** To me it seems like this could be a good paper with interesting ideas and solid experiments. However, the writing significantly holds it back and I think more time is required to allow  the authors to rewrite their paper. Furthermore, I think that there is a lot of useful information hidden in the appendix and it might make sense to focus the paper on the main idea (Section 3.1) and move other parts to the appendix to make more space.  As I do believe that this paper requires significant changes to the text I thus vote to reject.\n\nQUESTIONS:\nMost of my questions are already part of the weakness section.\n\n- **(Q1)** (Lie) UniConv are implemented as a $k$-th taylor approximation. What is the value for $k$? It seems to me like the ability to model long-range interactions is primarily dependent the value of $k$ and the number of layers $L$, so knowing this value seems important.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper proposed a new convolution operator UniConv on graphs and general tasks. Both graph convolution and general form convolution form are provided based on the group theory. With the toy experiment, it shows the advantages of the proposed method clearly. Based on both theoretical and experimental analysis, the proposed network is effective in long range graphs and heterophilic graphs.

STRENGTHS:
1. Theoretical analysis is conducted to show the effectiveness in handling oversmoothing and oversquashing issue.

2. The toy analysis is easy to understand and clearly shows the advantage.

WEAKNESSES:
1. Though the general form of convolution is provided. It is not evaluated in experiment. I will suggest putting these content in the appendix in this case. Otherwise, it is better to provide the corresponding experimental results.

2. The computational cost is higher compared to other methods, but there are no experimental results to show the important details.

QUESTIONS:
See weakness.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12