Revisiting Generalized p-Laplacian Regularized Framelet GCNs: Convergence, Energy Dynamic and as Non-Linear Diffusion
Authors: Dai Shi
Zhiqi Shao
Yi Guo
Qibin Zhao
Junbin Gao
Venue: Accepted by TMLR ()

Abstract: This paper presents a comprehensive theoretical analysis of the graph p-Laplacian regularized framelet network (pL-UFG) to establish a solid understanding of its properties. We conduct a convergence analysis on pL-UFG, addressing the gap in the understanding of its asymptotic behaviors. Further by investigating the generalized Dirichlet energy of pL-UFG, we demonstrate that the Dirichlet energy remains non-zero throughout convergence, ensuring the avoidance of over-smoothing issues. Additionally, we elucidate the energy dynamic perspective, highlighting the synergistic relationship between the implicit layer in pL-UFG and graph framelets. This synergy enhances the model's adaptability to both homophilic and heterophilic data. Notably, we reveal that pL-UFG can be interpreted as a generalized non-linear diffusion process, thereby bridging the gap between pL-UFG and differential equations on the graph. Importantly, these multifaceted analyses lead to unified conclusions that offer novel insights for understanding and implementing pL-UFG, as well as other graph neural network (GNN) models. Finally, based on our dynamic analysis, we propose two novel pL-UFG models with manually controlled energy dynamics. We demonstrate empirically and theoretically that our proposed models not only inherit the advantages of pL-UFG but also significantly reduce computational costs for training on large-scale graph datasets.

Published in Transactions on Machine Learning Research (02/2024)
Revisiting Generalized p-Laplacian Regularized Framelet GCNs: Convergence, Energy Dynamic and as Non-Linear Diffusion
Dai Shi* Western Sydney University
(u09::/90ssudeert.ssseesssyyyyy edu.
Zhiqi Shao" University of Sydney
Bbigl.shaa@ssydneyc.oom @syynneyyyy.ooom edu on
Yi Guo Western Sydney University
v.gnoo@uussterrsyddeeyy edu
Qibin Zhao AIP RIKEN
qibín. zh1o @riken.jpp
Junbin Gao University of Sydney
junbin. Burbin..gooon @sydney. edu
Reviewed on OpenReview: https: openreview. net/ forum? id= 94SSLPoFe7
Abstract
This paper presents a comprehensive theoretical analysis of the graph p-Laplacian reg- ularized framelet network (pI-UFG) to establish a solid understanding of its properties. We conduct a convergence analysis on pL-UFG, addressing the gap in understanding its asymptotic bchaviors. Further, by investigating the generalized Dirichlet energy of pL-UFG, remonstrate that the Dirichlet energy remains non-zero throughout, convergence, ensuring the avoidance of over-smoothing issues. Additionally, perspective, highlighting the synergistic relationship between the implicit layer in pL-UFG and graph framelets. This synergy enhances the model's adaptability to both homophilic and heterophilic data. Notably, we reveal that pI-UUFG can be interpreted as a generalized non-lineat diffuusio process, thereby bridging the gap between pL-UFG and differential equations on the graph. Importantly, these multifaceted analyses lead to unified conclusions that offer novel insights for understanding and implementing pL-UFG, as well as other graph neural network models. Finally, based our dynamic analysis, we propose two novel PI-UFG models with manually controlled energy dynamics. We demonstrale empirically and theoretically that our proposed models not only inherit the advantages of pL-UFG but also significantly reduce computational costs for training on largo-scale graph datasets.
elucidate the energy dynamic
1 Introduction
Graph neural networks (GNNs) have emerged as a popular tool for the representation learning on the graph-structured data (Wu et al. 2020). To enhance the learning power of GNNs, many attempts have been made by considering the propagation of GNNs via different aspects such optimization (Zhu et al. 2021 Wei et al. 2022), statistical test. (Xu et al. 2019) and gradient flow (Bodnar et al. 2022 Di Giovanni et al. 2023). In particular, treating GNNs propagation as an optimization manner allows one to assign different types of regularizers on the GNNs output so that the variation of the node features, usually measured by
Daa Shi and Zhiqi Shao with equal contributions

---

Published in Transactions on Machine Learning Research (02/2024)
so-called Dirichlet energy, can be properly constrained (Zhu et al. 2021 Chen et al. 2023 The underlying reason for this regularization operation is due to the recently identified computational issue of GNNs on different types of graphs, namely homophily and heterophily (Zheng et al. 2022a With the former, most of nodes are connected with those nodes with identical labels, and for the latter, it is not (Pei et al.) Accordingly, an ideal GNN shall be able to produce a rather smoother node features for homophily graph and more distinguishable node features when the input graph is heterophilic (Pei et al. 2019
2019
Bi et al.
2022).
Based on the above statement, proper design of the regularizer that is flexible enough to let GNN fit both two types of graphs naturally becomes the next challenge. Recent research Fu et al. (022) proposes new energy-based regularizer, namely the p-Laplacian- based regularizer, to the optimization of GNN and results in an iterative algorithu to approximate the so-called implicit layer induced from the solution of the regularized optimization problem. To engage a more flexible design of p-Laplacian GNN, Shao et al. (2023) further proposed p-Laplacian based graph framelet GNN (pI.UUFG) to assign the p-Laplacian based regularization acting on multiscale GNNs (c.g., graph framelet). While remarkable learning accuracy has been obscrved empirically, the underlying properties of the models proposed in (Shao et. al., 2023) are st.ill unclear. In this paper, our primary focus is pL-UFG (see Section 2 for the formulation). Our objective is to analyze PL-UFG from various perspectives, including convergence of its implicit layer, the model's asymptotic energy behavior, changes in the model's dynamics due to the implicit layer, and relationship with existing diffusion models. To the best of our knowledge, these aspects have not been thoroughly explored in the context of p- p-Laplacian based GNNs, leaving notable knowledge gaps. Accordingly, we summarize our contribution as follows:
We rigorously prove the convergence of PL-UFG, providing insights into the asymptotic behavior of the model. This analysis addresses a crucial gap in the understanding of GNN models regularized with p-Laplacian based energy regularizer.
We show that by assigning the proper values of two key model parameters (denoted μ and p) of PL-UFG based on our theoretical analysis, the (generalized) Dirichlet energy of the node feature produced from PL-UFG will never converge to 0; thus the inclusion of the implicit layer will prevent the model (graph framelet) from potential over-smoothing issue.
We demonstrate how the implicit layer in pL-UFG interacts with the energy dynamics of the graph framelet. Furthermore, show that pL-UFG can adapt to both homophily and hetcrophily graphs, enhancing its versatility and applicability.
We establish that the propagation mechanism within pL-UFG enables a discrete generalized non-linear graph diffusion. The conclusions based on our analysis from different perspectives are unified at the end of the paper, suggesting a promising framework for evaluating other GNNs.
Based on our theoretical results, we propose two generalized pL-UFG models with controlled model dynamics, namely pL-UFG low-frequency dominant model (pL-UFG-LFD) and PL-UFG high frequency dominant model (pL-UFG-HFD). We further show that with controllable model dynamics, the computational cost of pL.-UFG is largely reduced, making our proposed model capable of handling large-scale graph datasets.
We conduct extensive experiments to validate our thcoretical claims. The empirical results not only confirm pL-UFG's capability to handle both homophily and heterophily graphs but also demonstrate that our proposed models achieve comparable or superior classification accuracy with reduced computational cost. These findings are consistent across commonly tested and large-scale graph datasets.
The remaining sections of this paper are structured as follows. Section 2 presents fundamental nolations related to graphs, GNN models, graph framelets, and pL-UFG. In Section 3 we conduct a theoretical analysis on pI.-UFG, focusing on the aforementioned aspects. Specifically, Scction 3.1 presents the convergence analysis, while Section 3.2 examines the behavior of the p-Laplacian based implicit layer through a generalizod Dirichlet energy analysis. Furthermore, Section 3.3 demystifies the interaction between the implicit layer and
2

---

Published in Transactions on Machine Learning Research (02/2024)
graph framelets from energy dynamic perspective. We provide our proposed models (pL-UFG-LFD and pl-UUGGHFDD in Section 3.4 Lastly, in Section 3.5 we demonstrate that the iterative algorithm derived from the implicit layer is equivalent to generalized non-linear diffusion process on the graph. Additionally, in Section we further verify our theoretical claims by comprehensive numerical experiments. Lastly,
Section we summarize the findings of this paper and provide suggestions for future research directions.
2 Preliminaries
In this section, we provide the necessary notations and formulations utilized in this paper. We list the necessary notations with their meanings in Table
Table 1: Necessary Notations
Notations
Bricf Interprctation
H(G)
Heterophily index of a given graph G Initial node feature matrix
X
F() fo F
Feature representation on k-th layer of GNN model Individual row of F One operation acts each row of F
A
Normalized adjacency matrix Normalized Laplacian matrix
Graph weight matrix
W
Framelet decomposition matrix
I
Index set of all framelet decomposition matrices.
W
Learnable weight matrix in GNN models
W
Learnable weight matrices in defining generalized Dirichlet energy Feature propagation result for the pL-UFG defined in (Shao et al. 2023 N-dimensional vector for diagonal scaling (diag(0)) in Tramelet models. Generalized Dirichlet energy for node fcature induced from implicit layer
Y
EP (F)
We also provide essential background information on the developmental history before the formulation of certain models, serving as a concise introduction to the related works.
Graph, Graph Convolution and Graph Consistency We denote a weighted graph as 9 (V. E, W) with nodes set
U2.-.-,, of total N nodes, edge C V x V and graph adjacency matrix W, (vi, E E, else, wi.j = 0. The nodes feature matrix is X E Bxxx
where W [way] E RNxN and waj for G with each X; R the feature vector associated with node u. For matrix A, we denote its Throughout this paper, we will only focus on the undirect graph and use matrix A and W interchangeably for graph adjacency matrix The normalized graph Laplacian is defined as L I-I-D +W + IJD 4, where = diag(dl.,,..... .y,,.,, dx,x) is a diagonal degree matrix with d,,; Σ/= Wi,j for i = 1,..., N, and is the identity matrix. Let (A:) in decreasing order be all the cigenvalues of L, also known as graph spectra, and X; E [0, 2]. For any given graph, we let PT be the largest eigenvalue of L. Lastly, for any vector = [21, E Re, ||x||| = (2 x
transpose as AT, and we use [N] for set (1,
3 is the Lg-norm of x, and similarly, for any matrix M [ms,]], denote by |M] := [M]| (E,,, m the matrix Frobenius norm.
produces a layer-wise (node feature) propagation rule based on the information from the normalized adjacency matrix as:
Graph convolution network (GCN) Kipl & Welling 2016
((+11) = o (AF
(1)
We initially set. W the graph adjacency matrix while W a generic edge weight matrix align with the notations used in Pu
2022 Shao al. 2023)
3

---

Published in Transactions on Machine Learning Research (02/2024)
where F(R) is the embedded node feature, W()) the weight matrix for channel mixing (Bronstein et al. 20210, and any activation function such as sigmoid. The superscript (k) indicates the quantity associated with layer k, and F(o) X. We write À = D (W +I)D= i, the normalized adjacency matrix of G. The operation conducted in GCN before activation can be interpreted as localized filter by the graph Fourier transform, i.e., F(++1)- = U(, - AJU A) F((), where U, A are from the eigendecomposition == UAUT. In fact, UF is known as the Fourier transform of graph signals in F.
Over the development of GNNs, most of GNNs are designed under the homophily assumption in which connected (neighbouring) nodes are more likely to share the same label. The recent work by Zhuu et al. (2020) identifies that the general topology GNN fails to obtain outstanding results on the graphs with different class labels and dissimilar features in their connected nodes, such as the so-called heterophilic graphs. The definition of homophilic and heterophilic graphs are given by:
Definition 1 (Homophily and Heterophily). The homophily or heterophily of network is used to define the relationship between labels of connected nodes. The level of homophily of graph can be measured by the where HUG of U. E V that share the same label as U; i.e. yi = VJ- score H(g) close to corresponds to stronger homophily while a score H()) nearly 0 indicates stronger heterophily. We say that a graph is a homophilic (heterophâlic) graph if has stronger homophily (heterophily) or simply strong homophily (heterophily).
posilive score H(G) = DeeevlHe,,jeN..,,11/1
denoles the number of neighbours
Graph Framelet. The main target for this paper to explore is pL-UFG defined in (Shao et al. 2023 in
which the p-Laplacian-basel implicit layer is combined with so-called graph framelet or framelets in short. Framelets are a type of wavelet frames arising from signal processing that can be extended for analyzing graph signals. Dong (2117) developed tight framelets on graphs by approximating smooth functions with filtered Chebyshev polynomials. Graph framelets decompose graph signals and re-aggregating them effectively, as
shown in the study on graph noise reduction by [Zhou et al. (2021) Recently, Yang el. al. (2022) suggested a simple method for building more versatile and stable framelet families, known as Frausi-Faaeelts.... In this study, we will introduce graph framelets using the same architecture described in (Yang et al. 2022 To begin, we define the filtering functions for Onasi-framelets:
Definition 2. A set of R 1 positive functions F {go(E), (R..E)).. defined on the interval [o, || is considered as (a set of) Quasi- Frammeet scaling functions, if these functions adhere to the following identity condition:
go(E)² + .... +gR(E)² =1, VS E 10
(2)
The identily condition eq- (2 ensures a perfect reconstruction of signal from its spectral space to the spatial
space, see (Yang et al. 2022 for proof. Particularly
are interested in the scaling function set in which go
descents from to 0, i.c., (0) = 1 and (T()) O and gn ascends from 0 to 1, i.c., gn(()) = 0 and gn(x)) = 1. The purpose of setting these conditions is for go to regulate the highest frequency and for gR to control the lowest frequency, while the remaining functions govern the frequencies lying between them.
With a given set of framelet scaling functions, the so-called Quasi-Framele signal transformation can be defined by the following transformation matrices:
A Wo., = Ugol
2m+J/
go
2m
UT
(3)
A
Wr.o = Ugr(
JUT
for
r 1, R.
(4)
2m
A
A
Wr.e = Ugr 2
goo( 2m+t-1
go 2m
U
(5)
for =
=1 I,... R, a = I,..,,
where F is a given set of Quasi-Framelet functions satisfying eq. (2) and J 2 0 is given level on a graph g = EV,E) with normalized graph Laplacian L = U AU. Wo,J is defined as the product of J+1 Quasi- Framelet scaling functions go applied to the Laplacian spectra A at different scales. Wr is defined as gr 2m applied to spectra. A, where is the coarsest scale level which is the smallest value satisfying 2 Xx n.
4

---

Published in Transactions on Machine Learning Research (02/2024)
For S R and < < J. W,, is defined as the product of & one Quasi-Framen scaling functions 9r applied to spectra A at relevant scaling. Let W [Wo.J: Wi,0; ...;
scaling functions gu and
WR.ol be the stacked matrix. It can be proven that W"W = I, see (Yang et al. 2022), which provides a signal decomposition and reconstruction process based on W. This is referred to as the graph Qquasi-Framon Lransformation.
Since the computation of the Quasi-framelet transformation matrices requires the eigendecommposition of graph Laplacian, to reduce the computational cost, Chebyshev polynomials are used to approximate the Frameet transformation matrices. The approximated transformation matrices are defined by replacing (5) in eq- ---cq- 5 with Chebyshev polynomials (x(5) of fixed degree, which is typically set to 3. The can be approximated by,
Quasi-Frames transformation matrices defined in eq. (3) eq. 5
1 Wo,J 2 Tol 2m+J L) ... Tol 2m
L),
(6)
Wr.0 2 Tr( 2m Lt, for r = 1,.., ...,,R R.
(7)
LoTol 2m L) ..... To 1
1
Wr,t T 2 for
(o D) I,... R, l L,,...
Based on the approximated Quasi- Quasi-Framelet transformation defined above, two types of graph framelet convolutions have been developed recently:
The Spectral Framelet Models (Zheng et al. 2021 2022b
Yang et al. 2022 Shi et al.
2023a
p(++1) = (wT diag())FF :
Σ (r.))T
W. dia, (1) w(()
(9)
where 0r E IRN, (()) are learnable matrices for channel/featurs mixing, and I {(r,))
1,... 0,1 J), is the index set for all framelet decomposition matrices.
2. The Spatial Framclet Models (Chen et al. 2023
F(++1) =a | F W(l)) O.J + Σ A.W.A.,.F.. (P)) oe
(10)
The spectral framclet models conduct framelet decomposition and reconstruction on the spectral domain of the graph. Clearly 0r, E N can be interpreted as the frequency filters, given that a framelet system provides a perfect reconstruction on the imput graph signal (i.e., WTW=1 I). Instead of frequency domain filtering, the spatial framelet models implement the framelet-basod propagation via the spatial (graph adjacency) domain. There is a major difference between the two schemes. In the spectral framelet methods, the weight matrix (kk) is shared across different (filtered) frequency domains, while in the spatial framelet. methods, an individual weight matrix (k) is applicd to each (filtrred) spatial domain to produce the graph convolution. It worth noting that the theoretical exploration of the learning advatage of graph framelet models, as well
r.t
as the difference between two types of framelet models, have been studied in (Han al., 2022 Zheng et al. 2021
Chen et al. 2023 Shi et al. 2023b
Generalized p-Laplacian Regularized Framelet GCN. In this part, provide several additional definitions to formulate the model (pL-UFF) that we are interested in analyzing. As a generalized framclet model incorporating the so-called p-Laplacian energy regularizer, the pL-UFG, as we are going to define it later, has shown great flexibility terms of adapting different types of graphs (i.e., homophily and heterophily) by efficiently adjusting the penalty strength from the regularizer, resulted in superior learning performance across various benchmark datasets (Shao et al. 2023). We started by defining the p-Laplace operator on the graph follows.
5

---

Published in Transactions on Machine Learning Research (02/2024)
Definition (Graph p-Laplacian). Given graph (V, E, W) and multiple channel signal function F E Fv, the graph p-Laplacion is operator Ap Fv Fv, defined by:
F= = - dI for p p21. 1.
(11)
where div and V the graph gradient and divergence operators power over the node gradient VI
Fu et al. 2022 || ||p-2 is element-wise
The corresponding p-Dirichlet form can be denoted as:
(,(F)
1
Σ (EE
Wi Willf di
fj
Wi.j f di,i
(12)
where we adopt the definition of p-norm (Fu al. 2022 It is difficult verify that once we set p 2, recover the graph Dirichlet energy (Zhou & Schölkopf, 2005) that is widely used to mcasure the difference between node features along the GNN propagation process.
Remark 1 (Dirichlet Energy, Graph Homophily and Heterophily). Graph Dirichlet energy (Fu al. 2022 Bronstein et al.) 2021 has become commonly applied measure of variation between node features via GNNss It has been shown that once the graph is highly heterophilic where the connected nodes are not likely to share identical labels, one may prefer GNNs that exhibit nodes feature sharpening effect, thus increasing Dirichlet energy, such that the final classification output of the connected nodes from these GNNs tend to be different. Whereas, when the graph is highly homophilic, a smoothing effect (thus a decrease of Dirichlet energy) preferred.
Shao et al. 2023 further generalized the
p- D-Driichlet form in eq. 12 as:
Sp(F)
Σ
V
F(E
(WiRS))EE
17
Σ WEV
Σ 1ym0e
F(E.DDI
2
Σ NVV
NVwFF(u;))
(13)
where by ~ U; stands for the node uy that connected node 9 and F(.) F(i, (() the node gradient vector for each node i and || Ipp is the vector p-norm. Moreover, we can further generalize the regularizer (F) by considering any positive convex function & as:
e is
1 Σ 2
(F)
V w F(w:)l))
(14)
BEV
There are many choices of & and When ({E)= = S", we recover the p-Laplacian regularizer. Interestingly, by setting (fE) = =², we recover the so-called Tikhonov regularization which is frequently applied in image processing. When Φ(E) = E, ie. identity map written as id, and p =1,, S (F) becomes the classic total variation regularization. Last but not the least, ({E) = r2 log(1 + 2/rr/²² gives nonlinear diffusion. We note that there are many other choices on the form of Φ. In this paper we will only focus on those mentioned in (Shao al. 2023) (i.e., the smooth ones). As result, the flexible design of the p-Laplacian based energy regularizer in eq. [14] provides different penalty strengths in regularizing the node features propagated from GNNs.
Accordingly, the regularization problem proposed in Shao el al. 2023 is:
= arg min S. (F) + F T diag (F)
(15)
We provide detailed formulation graph p-pallacian Appendix A.I.1

---

Published in Transactions on Machine Learning Research (02/2024)
where let Y WT diag(O)WF stands for the node feature generated by the spectral framelet models
without activation This is the implicit layer proposed in (Shao et al. 2023). As the optimization problem
defined in eq. 115 does not have closed-form solution when * 2, an iterative algorithm is developed
in (Shao et al. 2023 to address this issuc. The justification is summarized by the following proposition
(Theorem Shao et al.
2023
Proposition 1. For given positive convez function ((E) (o(E) 2 on the domain R+), define
M.j.
Widd
V
F(i.JMI
o' F(c)))))
d'(IV W F(
(16)
2
Vw (z.
(VwF(())
Mijj
Oiii
=1/
Σ
+ 2/u
Bs = 2poss
(17)
dif
Ny
and denote the matrices M [Ma a diag diag(a11. 0111 ANN) and B = diag(311, ... BNN). Then problem 15 can be solved by the following message passing process
F(++1)_ ( 1/2 1/2 D 1/2 + g
(18)
with initial value, e.g., calculated according to eq. 16 with ((k)
F = 0 Y. Note, denotes the discrete time indez (iteration) and M is
Finally, in this paper, we call the iterative algorithm defined in eq. 18 for realizing the implicit layer defined by the solution of problem eq. [15 as the pL-UFG model. Although remarkable performance has been observed from pI-UFG, there are still some key properties of the model that. require to be theoretically explored. For example, the convergence of the iterative algorithm eq. (18) for the implicit layer in eq. (15) and how the implicit layer changes and interacts with the energy dynamic of the original framelet, what is the relationship between the propagation within the implicit layer and other propagations such as diffusion node features. We will explicitly show our theoretical results in the coming sections.
3 Theoretical Analysis of the pL-UFG
In this section, we show a detailed analysis of the convergence (Section 3.1 and energy behavior (Section 3.2 of the iterative algorithm in solving the implicit layer
presented in eq- 18). In addition, we will also present some results regarding the interaction between the implicit layer and graph framelet in Section 3.3 via the energy dynamic aspect based on the conclusion from Section 3.2 Lastly, in Section 3.5) we will verify that the iterative algorithm induced from the p-Laplacian implicit layer admits a discretized non-linear diffusion process, thereby connecting the discrete iterative algorithm to the differential equations on the graph.
First, we consider the form of matrix M in eq.
18
Write
F + V
(i (F) 2 [[eF(()))
(19)
F(;))l
Myy can be simplified as
M1 = Sh (F) u V F(i.)))
(20)
(F) is bounded as shown in the following lemma.
Lemma 1. For the given p used in St (see eq.
14)), assume that
g((
EP-1 S C,
(21)
for a suitable constand C, then we have (F) C.
7

---

Published in Transactions on Machine Learning Research (02/2024)
The proof is trivial thus omit it here. In the sequel, we use (F) for Si, (F) instead.
The assumed condition in (21) in Lemma is appropriate to ensure (F)(F) is bounded. This condition is satisfied for all Φ functions used in (Shao et al. 2023
Remark 2.
When (() <" (1 S oo) and take the same p into St (F)), then the resulting (F) is bounded for all such p:
When ((E) = g² and for any 0 < p I 2 in Sp (F), then the resulting Si (i,(F) is bounded when F is bounded. This is because 5'(5)
{--1 = 2
which is bounded for finite 5 > 0;
EF
When ((E) 3, then d'(E) (--)
S
indicating Gi is bounded for all <<pS
When d({) v + 2 - e, have (E(E) all 0 < S 2;
Therefore Ci (F) is bounded for C T
When <(5) = 2 log( (1 + )) the result of ( bounded for all 0 < 2
yields p2
2 1+ got
48
Hence (i,(F) remain e
In summary, for all forms of & we included in the model, (F) is bounded with the appropriate choice of p. The boundedness of (1,(F)
(F) from Lemma 1 is useful in the following convergence analysis.
3.1 Convergence Analysis of pL-UFG
We show the iterative algorithm presented in eq. 18 will converge with a suitable choice of μ.. We further nole that although the format of Theorem is similar to Theorem in (Fu et al. 2022), our message passing scheme presented in eq. (18) different compared to the one defined in (Fu al 2022) via the forms of M, and B. In fact, the model defined in (Fu et al. 2022 can be considered as special case where Φ({) 8".
As a generalization of the model proposed in (Fuu et al. 2022 we provide a uniform convergence analysis for the pL- UFG.
Theorem (Shrinking Property of the Proposed Model Iteration). Given a graph G(V, E W) with node features X, if (*),,, gg(),, (() and Fk) are updated according to eq. [1), then there erist some real positive value μ, which depends on the input graph (9, X) and the quantity of p, updated in each iteration, such that: (P(tt)) < (g(F(()),
where C$(F) = :
We include detailed proof in Appendix A.2.1 Theorem 1 shows that with an appropriately chosen value of μ, the iteration scheme eq. (18) for the implicit layer eq. (15) is guaranteed to decrease the objective function values (or not increasing). This inspires us to explore further the variation of the node feature produced from the implicit layer asymptotically. Recall that to measure the difference between node features, one common choice is to analyze its Dirichlet energy, which is initially considered in the setting p = 2 in eq. 12). is known that the Dirichlet energy of the node feature tends to approach after a sufficiently large number of iterations in many GNN models (Kipf & Welling 2016 Wu al. 2020 Chamberlain et al., 2021a: Di Giovanni al., 2023), known the over-smoothing problem. However, as will show in the next. section, by taking large or small p, the iteration from the implicit layer will always lift up the Dirichlet energy of the node features, and the over-smoothing issue can be resolved completely in pL-UFG.
Remark 3. Additionally, Theorem guarantces that the objective C3 is bounded, which means all the iterates generated from eq. (18) are bounded (as [ - Y is bounded). Hence, for the sequence ( lk=1 there must be an accumulated point F", such that there is convergent sub-sequence (F(...))oo jo0 Accordingly, the iteration produces a weak convergence sequence eq.
18 in the sense of sub-convergent.
Remark 4. In practice, the implicit layer eq. (15) replaced by a given number (c.g., 20 in our experiments) of iteration eq. (18). Theorem ensures the objective value is not increasing. Thus this shrinkage property is more important than the convergence in practice.
8

---

Published in Transactions on Machine Learning Research (02/2024)
3.2 Energy Behavior of the pL-UFG
In this section, show the energy behavior of the p-Laplacian based implicit layer. Specifically, are interested in analyzing the property of the generalized Dirichlet energy defined in (Bronstein et al. 2021 We start by denoting generalized graph convolution as follows:
F(k+T) FF(k) To (-F(K) AF W(t) p()W()))
(22)
where (2)), ( and W(k E Rexe act on each node feature vector independent and perform channel mixing. When setting (2 * 0, we have the anisotropic instance of GraphSAGE (Xu et al. 2019). To quantify the quality
and f2(k) wik) it returns to GCN (Kipf & Welling 2016). Additionally, by
of the node features generated by eq. 22 specifically, Bronstein et al. (0021) considered new class of energy defined below,
1 N E(F)
N Σ
f. Sif.))
(f, Wf,) + (,j=1
(23)
2
in which (0(0) (F, (()) serves function of that induces the source term from F or F(O), It is worth noting that
by setting f W L and (0) 0, recover the classic Dirichlet energy when setting p 2 in eq. 12
that is, E(F) = 2 Σ DE
f
Mi. fg
Additionally, when we set (( (F, F()) = Σ, (f. Wt
de.
oq. 23 can be rewritten as:
1 vec
W A'vece)(F)) + (W & IN Lx)
E(F)
S
IN
(24)
18) produces the node feature F(++1) according to the edge diffusion aD -D/1/DMD 1/2 on [(4) and the scaled source term 2)x(((0) where F(0) can be set to Y. To be specific, in 24). we set Ω= -= W I. and replace the edge diffusion Á with aD aD-1/2MD 11/2 1/2 and set the identity matrix IN in the residual term to be the diagonal matrix 2pa. Finally, we propose
Recall that eq.
Definition (The Generalized Dirichlet Energy). Based on the previous notation setting, the generalized Dirichlet energy for the node features F(++1) in eq. 18 is:
EP (F(++1)) = -(vec(p(tt)))
1 IN - 1. 8 (a(++ ((/MM D-1/2) vec(F((+11) + (I. @ (
2 1. 1681
(25)
where the superscript aPF " is short for p-Lpllccian based framelet models.
It is worth noting that the generalized Dirichlet energy defined in eq. 25
is dynamic along the iterative We are now able to analyze the energy (E (F)) behavior of the pL-UFG, concluded as the following proposition.
layers due to the non-linear nature of the implicit layer defined in eq. 15
Proposition 2 (Energy Behavior). Assume G is connected, unweighted, and undirected. There erists sufficiently large value of or small value of p such that EPF (F) will slay away above O at each ilerutive layer and increases with the increase of μ or the decrease of p.
We leave the detailed proof in Appendix A.2.2 models, the p-Laplacian based implicit layer will not generate identical node features across graph nodes, and thus the so-called over-smoothing issue will not appear. Furthermore, it is worth noting that the result from Proposition 2 provides the theoretical justification of the empirical observations in (Shao al. 2023). where large value of μ or small value of p is suitable for fitting heterophily datasets which commonly require the output of GNN to have higher Dirichlet energy.
Proposition 2 shows that, for any of our framelet convolution
9

---

Published in Transactions on Machine Learning Research (02/2024)
Remark 5 (Regarding to the quantity of p). The conclusion of Proposition 2 is under sufficient large of I or small of p. However, it is well-known that the quantity of p cannot be set as arbitrary values, and in fact, it is necessary to have p 2 1 so that the iteration for the solution of the optimization problem defined in eq. (15) can converge. Therefore, it is not hard to see that the effect of p is weaker than f in terms of analyzing the asymptotic energy behavior of the model (i.e., via eq. 44) in Appendix (...2).. Accordingly, in practice, one shall prefer to let μ be the major target and p be the sub-target to adjust for fitting pI.-UFG to different types of graphs. Without loss of generality, in the sequel, when we analyze the property of the model with conditions j and p. we mainly target the effect from J4, and one can check μ and p are in the opposite effect on the model.
3.3 Discussion on Dynamic Interactions
The results on the energy behavior of the p-Laplacian implicit layer, as discussed in the last section, encourage further exploration into the interactions between the implicit layer and graph framelets. The dynamics of energy in graph framelets have become a popular topic in recent works (Shi et al. 2023a Han et al. 2022] 2023). The studies by Han et al. (2022) and Shi et al. (2023a) demonstrate that, under mild conditions, the propagation dynamics of graph framelets can be dominated by one of its frequency domains, as exemplified by the Graph Neural Network (GNN) dow-frequencyy-ooinnnee (TFD) and High-firqquency-ominnt
(HFD).
Taking spectral framelet convolution eq. 19 wwtt Haar-type filter (i.e. R R= 1 in the case of scaling function set) as an example, one can manually set 0,.1 = 11v and 01.1 = 01N, and if f E [0, 1), the spectral framelet convolution is LFD, meaning that although the feature propagation via high-frequency domain remains to sharpen node features, the mainstream of framelet is dominated by the dynamic via low-frequency domain in which node features are homogenized by the low-pass filter. Therefore, asymptotically all node features will still become identícal, resulting in over-smoothing issues. However, this situation can be reversed by sctting 8x > 1, and the resulting HFD dynamic in the framelet ensure that all features are sharpened in the whole propagation process. Therefore, the over-smoothing issue will not appear. It is worth noting that there are many other settings that can induce LFD/HFF dynamic of framelet, our example is here as an illustration purpose.
Now recall that based on Definition 1 G is homophilic, connected nodes are more likely to share the same label, in this case, one may prefer framelet model to induce LFD dynamic to enhance smoothing. On the other hand, if G is heterophilic, the model is expected to induce an HFD dynamic so that even in the adjacent nodes, their predicted labels still tend to be different. Based on the aforementioned settings 0, one can observe that graph framelet can induce both LFD and HFD dynamics so that automatically capable of fitting both types of graphs. We strictly formulate the LFD and HFD dynamic of framelet in Appendix A...2 Together with the result in Proposition 2 it can be verified that if the framelet is LFD, the additional p- L-Lalaccian implicit layer will ensure the Dirichlet energy of the node feature remains nonzero, thus preventing the model from the over-smoohhing issuc. On the other hand, if the framelet is HFD, node features are initially sharpened from the framelet and further serve as a source term in the implicit. layer (i.e., 2) This setting provides the potential to induce further sharpening of the node feature via the implicit layer, resulting in an "enhanced" HFD dynamic on the framelet to further fit heterophilic graphs. We empirically verify this observation in Section
3.4 Proposed Model with Controlled Dynamics
Based on the aforementioned conclusions regarding energy behavior and the interaction between the implicit layer and framelet's energy dynamics, it becomes evident that irrespective of the homophily index of any given graph input, one can readily apply the condition of 0(s) in Proposition B to facilitate the adaptation of the PL-UFG model to the input graph by simply adjusting the quantities of μ and p. This adjustment significantly roduces the training cost of the graph framelet. For instance, consider the case of employing a Haar lype frame with l == 1, where we have only one low-pass and one high-pass domain. In this scenario, the trainable matrices for this model are 0p.1, 01.1, and W. Based on our conclusions, we can manually set both 0 and 11, to our requested quantities, thereby inducing either LFD or HFD. Consequently, the only remaining training cost. is associated with W, leading to a large reduction in the overall training cost.
10

---

Published in Transactions on Machine Learning Research (02/2024)
00.1 1x 900.1
Y
PL-UPC-LFD
High- Pass llovel
EE [0,
LFD
Y
HFD
PI-UFG-HFD
High- Pass
0,,1
(lovel
00..1
Framelet (Hear)
HFD
Node feature after framelet
Small 1 Large
Graph gradient based implicit layer.
Greater LFD
F
If
Large
(G (6 10
Small p
Graph Gradient
Greater HFD
Layer Output
Figure Illustration of the working flow of PL-UFG-LFD and PL-UFG-HFD under the Haar type frame with The input graph features are first decomposed onto two frequency domains and further filtered by the diagonal matrix 0,1 and 01,1- With controlled model dynamics from Proposition i.c., 0., 1N and 01,1 00,,, framelet can induce both LFD and HFD dynamics resulting different level of Dirichlet energy of the produced node features. It is straightforward to check that when the framelet is LFD, the level of node Dirichlet energy is less than its HFD counterpart. The generated node features from the graph framelet is then inputted into p-Laplacian (with graph gradient as one component) based implicit layer.
while preserving the model's capability of handling both types of graphs. Accordingly, we proposed two additional pL-UFG variants with controlled model dynamics, namely PL-UFG-LFD and PL-UFG-HFD. More explicitly, the propagation of graph framelet with controlled dynamic takes the form:
((+11) =a (Wo diag(IN Wo, F W di) F(b) W)
After that, the output node features will be propagated through certain iterative layers as defined in eq. 18 for the implicit layer eq. (15). and the resulting node feature will be forwarded to the next graph framelet convolution and implicit layer propagation. To properly represent the Dirichlet energy of node features, we borrow the concept of electronic orbital energy levels in Figure The shaded outermost eloctrons correspond to higher energy levels, which be analogously interpreted higher variations in node features. Conversely, the closer the electrons are to the nucleus, the lower their energy levels, indicating lower variations in node features.
3.5 Equivalence to Discrete Non-Linear Diffusion
Diffusion on graph has gained its popularity recently (Chamberlain et al. 2021b Thorpe el. al. 2022 Han
et al. 2023) by providing a framework (i.e., PDE) understand the GNNs architecture and as principled way to develop a broad class of new methods. To the best of knowledge, although the GNNs induced from discretized linear diffusion graph [Chamberlain et al. 2021b Thorpe et al. 2022 Shi et al. 2024
have been intensively explored, models built from non-linear diffusion have not attracted much attention in
general
In this section, we aim to verify that the iteration eq. 18 admits a (discretized) nonlinear diffusson
For details including the relationship hetween GNN dynamic and continuous diffusion process, please check CChamberlan
2021al Han et 2023
11

---

Published in Transactions on Machine Learning Research (02/2024)
with source term. To see this, recall that the p-Laplacian operator defined in eq.
11 has the form:
P _ div(
for p 1.
(26)
Plugging in the definition of graph gradient and divergence (Specifically defined in eq. the above equation, one can compactly write out the form of p-Laplacian as:
(30 and eq-
32
into
(A) = Σ
Wi d
(V (V
Wi f.
Wi d3s"
fj
(27)
de.
V
If we treat the iteration equation eq. process on the graph, we have
18 as the Euler discretization of one continuous non non-linear diffusion
F(k+1) FF())
(1) D-DD-11 1/2 1/2 (k) D 1/2p(k) - FF()) + 3(x) Y. p(() + g (a) ( 1/2 ( D 1/2
- D-1/2
(28)
We set T = for the rest the analysis for convenience reasons. With all these setups, we summarize our results in the following:
Lemma 2 (Non-Linear Diffusion). Assuming g is connected, the forword Euler scheme presented in eq.
28
admits a generalized non-linear diffusson on the groph. Specifically, we have:
(a 1 D--DDD--- 1/2
D-12 -1) 111111111 F(() F) + B = (p= + 2ua (F) + 2ua (k) F())
(1)
(29)
We leave the proof of the Lemma in Appendix A.2.3 Based on the conclusion of Lemma 2 it is clear that the propagation via the p-Laplacian implicit layer admits scaled non-linear diffusion with two source terms. We note that the form of non-linear diffusion coincides with the one developed in (Chen al., 2022). However, in Chen et al. 22022; the linear operator is assigned via the calculation of graph Laplacian whereas in our model, the transformation acts over the whole p-Laplacian. Finally, it is worth noting that the conclusion in Lemma 2 can be transferred to the implicit scheme [
We omit it here.
Remark 6. With sufficiently large or small p, one can check that the strength of the diffusion, i.e. div (p-2)p(()) is diluted. Once two source terms (k) + (+) dominant the whole process, the generated node features approach to DF + F()),, which suggests a framelet together with two source terms. The first term can be treated as the degree normalization of the node features from the last layer and the second term simply maintains the initial feature embedding. Furthermore, this observation suggests our conclusion described in Section 3.3 and lastly, the conclusion from Lemma can be unified and eventually forms well-defined framework in assossing and understanding the property of PL.-UFG.
the energy behavior of pL-UFG (Proposition 2); the interaction within PL-UFG
4 Experiment
Experiment outlines In this section, we present comprehensive experimental results on the claims that we made from the theoretical aspects of our model. All experiments were conducted Py Torch on NVIDIA Tesla V100 GPU with 5,120 CUDA cores and 16GB HBM2 mounted on an HPC cluster. In addition, for the sake of convenience, we listed the summary of cach experimental section as follows:
In Section 4.1 tintroophlic/hhmmpphhe graphs, and the results are almost invariant to the choice of p.
we show how a sufficiently large/smal μ can affect the model's performance on
"With duplication of terminology, here the "implicit" refers the implicit scheme (lee., backward propagation) in the training the diffusion model.
12

---

Published in Transactions on Machine Learning Research (02/2024)
In Section 4.2 we show some tests regarding the results of the model's dynamics. Specifically, we verified the observations in Section 3.3 with controlled model dynamics (quantity of 0 ) of framelet. to illustrate how the p-Laplacian based implicit layer interacts with the framelet model.
In Section 4.3 we test the performances of PI--UFG-LFD and PL-UFGG-HD via real-world graph benchmarks versus various baseline models. Furthermore, as these two controllable pL-UFG models largely reduced the computational cost (as we claimed in Section 3.4), we show PL-UFG-LFD and pL-UFG-HFD can even handle the large-scale graph datasets and achieve remarkable learning accuracies.
Hyper-paramettr tuning We applied exactly the same hyper-parameter tunning strategy as (Shao et al, 2023 to make a fair comparison. In terms of the settings for graph framelets, the framelet type is fixed as Haar Yang et al. 2022 ) and the level J is set to 1. The dilation scale s E (1, 1.5, 2,3, 6), and for n, the degree of Chebyshev polynomial approximation is drawn from {2, 3, 7}. It is worth noting that in graph framelets, the Chebyshev polynomial is utilized for approximating the spectral filtering of the Laplacian eigenvalues. Thus, a d-degree polynomial approximates d-hop neighbouring information of each node of the graph. Therefore, when the input graph is heterophilic, one may prefer to have a relatively larger as node labels tend to be different. between directly connected (I-hop) nodes.
4.1 Synthetic Experiment on Variation of
Setup In this section, we show how a sufficiently large/small of μ can affect the model's performance on hetero/homope hoooophiio graphs. In order to make fair comparison, all the parameters of pL.-UFG followed the settings included in (Shao al., 2023). For this test, we selected two datasets: Cora (heterophilic index: 0.825, 2708 nodes and 5278 edges) and Wisconsin (heterophilic index: 0.15, 499 nodes and 1703 edges) from https: http://wwwpppppppp../ pyg. org/ We assigned the quantity of p {1, 2, 2.5) combined with set of μ = f0.1, 0.5, 1, 5, 10, 20, 30, 50, 70). The number of epochs was set to 200, and the test accuracy (in %) was obtained as the average test accuracy of 10 runs.
Results and Discussion The experimental results are presented in Figure 2 When examining the results obtained through the homophily graph (Figure 2ap. it is apparent that all variants of pL-UFGs achieved the best performance when j = 0.1, which is the minimum value of μ.. As the value of ju increased, the learning accuracy decreased. This suggests that a larger sharpening effect was induced by the model, as stated in Section 3.3 and Proposition [ causing pL-UFGs to incorporate higher amounts of Dirichlet energy into their generated node features. Consequently, pL-UFGs are better suited for adapting to heterophily graphs. This observation is further supported by the results in Figure 26 where all pL-UFG variants achieved their optimal performance with a sufficiently large μe when the input graph is heterophilic.
Additional interesting observation on the above result is despite the fact that all model variants demonstrated superior learning outcomes on both homophilic and heterophilic graphs when assigned sufficiently large or small values of μu: it can be observed that when the quantity of p is small, pL-UFG requires a smaller value of y to fit the heterophily graph (blue line in Figure Zbb. On the other hand, when the models have relatively large value of (i.e., p 2.5), it is obvious that these models yielded the most robust results when there is an increase of (red line in Figure 2a.. These phenomena further support the notion that p and μ exhibit opposite effects on the model's energy behavior as well as its adaptation to homophily and heterophily graphs.
4.2 Synthetic Experiment on Testing of Model's Dynamics
Now, we take step ahead. Based on the observation in Section 3.3 the inclusion of p p-Laplacian based implicit layer has the potential for further enhancing framelet's 's LFD and HFD dynamics. This suggests that one can control the entries of 0 based on the conditions provided in Proposition 3 and by only changing the quantity of μ and p to test the model's adaption power on both homophily and heterophily graphs. Therefore, in this section, we show how a (dynamic) controlled framelet model can be further enhancod by the assistant from the p-Laplacian regularizer. Similarly, we applied the same setting to the experiments in (Sha et al. 2023
13

---

Published in Transactions on Machine Learning Research (02/2024)
2-33
073
(a) Accuracy Cora via different combinations of (b) Accuracy Wisconsin via different combinations of and
and p.
Figure 2: Performance of pI-UFG with various combinations of the values of and p.
Setup and Results To verify the claims on in Section 3.3 we deployed the same settings mentioned in Proposition Specifically, we utilized Haar frame with = and set 0o. = IN, 0o.1 = ANx. For heterophilic graphs (Wisconsin), f 2, and for the homophilic graph (Cora), experiment is presented in Figure g Similar to the results observed from Section 4.1 it is shown that when the relatively large quantity of ju is assigned, the model's capability of adapting homophily teuoohiilyhhteeeee /heterophlly graph decreased picreased/inccressed. /increased.. This directly verifies that the p-Laplacian based implicit layer interacts and further enhances the (controlled) dynamic of the framelet by the value of p and in terms of adaptation.
0.2. The result of the
Ascuge Accursyyii with Changing on Zora unde ur hursets
(a) Accuracy on Cora
(b) Accuracy on Wisconsin
Figure 3: Average Accuracy(% (%) with Changing y and p under (manually fixed) LFD/HFD framelet models. All framelet model in Figure 3a are LFD dynamic with 0o.1 = IN, 01,1 = 01N, 0.2. On Figure 3b all framclet models are HFD with f,,1 IN. 811 01N, 0 2.
4.3 Real-world Node Classification and Scalability
Previous synthetic numerical results show predictable performance of both PL-UFG-LFD and PL-UFG-HFD. In this section, we present the learning accuracy of our proposed models via real-world homophily and heterophily graphs. Similarly, we deployed the same experimental setting from (Shao et al. 2023). In addition, to verify the claim in Remark 3.4, we tested our proposod model via a large-scale graph datasct (ogbn-arxiv) show the proposed model's scalability, which is rarely explored. We include the summary statistics of the datasets in Table 2 All datasets are split according to (Hamilton al. 2017
14

---

Published in Transactions on Machine Learning Research (02/2024)
Table 2: Statistics of the datasets, H() represent the level of homophily overall benchmark datasets.
Dalasets
Class
Peature
Node
Edge
H(
Cora
7
1433
2708
5278
0.825
CiteSeer
6
3703
3327
4552
0.717
PubMed
500
19717
44324
0.792
Computers
10
767
13381
245778
0.802
Photo
8
745
7487
119043
0.849
CS
15
6805
18333
81894
0.832
Physics
5
8415
34493
247962
0.915
Arxiv
23
128
169343
1166243
0.681
Chameleon
5
2325
2277
31371
0.247
Squirrel
5
2089
5201
198353
0.216
Actor
5
932
7600
26659
0.221
Wisconsin
5
251
499
1703
0.150
Texas
5
1703
183
279
0.097
Cornell
5
1703
183
277
0.386
For the settings of μ, and 0 within pL-UFG-LFD and PL-UFG-HFD, we assigned μ 0.51, 1, 1,2, = 2,, and 0 = {0.2, 0.5, 0.8} for PL-UFG-LFD in order to fit the homophily graphs, and for PL-UFG-HFD, we assigned μ = (10, 20, 30), = {1. 1.5, 2, 2.0. 2.5} and = {5, 7.5, 10] for heterophily graphs. The learning accuracy presented in Table 3 and 4 Furthermore, rather than only reporting the average accuracy and related standard deviation, computed the 95% confidence interval under f-distribution for the highest learning accuracy of the baselines and mark to model's learning accuracy if it is outside the confidence interval.
further verify the significance of the improvement, we also
We include a brief introduction to the baseline models used in this experiment:
MLP: Standard feed-forward multiple layer perceptron.
GCN Kiip & Welling 2016): GCN the first of kind to implement linear approximation spectral graph convolutions.
SGC (Wu et al. 2019 : SGC reduces GCNs' complexity by removing nonlincarities and collapsing
weight matrices between consecutive layers. Thus serves as a more powerful and efficient GNN bascline.
GAT (Veličković et al. 2018 : GAT generates one attention coefficient matrix that is element-wisely
multiplied on the graph adjacency matrix according to the node feature-based attention mechanism via each layer to propagate node features via the relative importance between them.
JKNet Xu et al. ranges, facilitating enhanced structure-aware representation for individual nodes.
2018): JKNet offers the capability to adaptively exploit diverse neighbourhood
APPNP leverages personalized PageRank to disentangle the neural network from the propagation scheme, thereby merging GNN functionality.
APPNP Gastteiger el al.
2019h:
GPRGNN (Chien et al. 2021] : The GPRGNN architecture dynamically learns General Pagerank (GPR) weights to optimize the extraction of node features and topological information from graph, irrespective of the level of homophily present.
p-GNN Fu et al. 2022 :p- GN-GNN a p-aalaccian based graph neural network model that incorporates message-passing mechanism derived from discrete regularization framework. To make fair comparison, we test the P-GNN model with different quantilies of
UFG [Zheng 2022b): UFG, class of GNNs built upon framelet transforms utilizes framelet decomposition to effectively merge graph features into low-pass and high-pass spectra.
15

---

Published in Transactions on Machine Learning Research (02/2024)
(hao et al 2023 pL-UFG employs p-Laplacian-based implicit layer enhance the adaptability of multi-scale graph convolution networks (i.e., UFG) to filter-bsedd domains, effectively improving the model's adaptation to both homophily and heterophily graphs. Furthermore, as two types of pL-UFG models are proposed in (Shao et al.) 2023), we test both two pL-UFG variants as our baseline models. For more details including the precise formulation of the model, please check (Shao al., 2023).
PL-UFG
Table 3: Test accuracy (%) on homophilic graphs, the top learning accuracy highlighted in bold and the second accuracy is underlined. The term OOM means out of memory.
Method
Cora
CiteSeer
PubMed
Computers
Photos
CS
Physics
Arxiv
MLP
66.04+1.11
68.99+00.48
82.03+0.22
71.89+5.36
86.11-1.35
93.50+0.24
94.56+0.11
35.50++0.78
GCN
84.72=0.38
75.01+1446
83.19+0.13
78.82+1.87
90.00=1.49
93.00=0.12
95.55+0.09
70.07+0.79
SGC
83.79:10.37
73.52.10.80
75.9210.26
77.56.10.88 88
86.44-10.35
92.18.1.22
94.9910.13
71.01:0.30
GAT
1..37+1113
74.80+1.00
83.92+0.28
78.68+22.99
89.63+1.77
92.57 +0.14
13.13+
0OM
JKNet
83.69:10.71
74.49.10.74
82.59.0.34
69.32.13.94
1.112:1112
91.11110.22
94.45.10.33
00M
APPNP
83.69=0.771
75.84+0.64
80.42+0.29
73.73+2.49
87.03=0.95
91.52=0.14
94.71+0.11
0OM
GPRGNN
83.7920.93
75.9410.65
82.32+0.25
74.26:1..4 94
88.69=1322
91.89 10.08
94.8510.23
00M
UFG
80.64::0.74
73.30
81.52:0..00
66.39:16.09
86.60-4.69
95.27
95.77:10.04
71.08:10499
PGNNL
84.21=0.91
75.38+0.82
84.34+0.33
81,22+2.62
87.64=5.05
94.88=0.12
96.15++0112
00M
PGNNL5
84.4220.71
75.4410.98
84.4810.21
82.68:1.115
91.83
94.13210088
96.1440.08
00M
PGNN2.
84.74+00.67
75.62+1.07
84.25 +..35
03.40
91.71
94.28+00.10
96.03++0.07
00M
PGNN25
84.48=0.77
75.22+0.73
83.94=0.47
12.91+1.34
91.41=0.66
93.40=0.07
95.75=#0.05
0OM
1
84.5410.62
75.8810.60
85.5610.18
82.0712.78
85.57.19.99
95.0310.22
16.10.00.06
70.2819.13
PLAURGIAS
84.96+0.38
76.04+0.85
85.59+0.18
85.04+1.06
92.92-0.37
95.03+0.22
96.27+0.06
71.25+8.37
2L-UFG120
85.20=0.42
76.12
85.59
85.26+1.15
92.65=0.65
94.77=0027
96.04+0.07
00M
2L-UFG125
85.30.10.60
76.11
85.5410.18
15.10.1
91.49-11.29
94.86.10.14
05.9610.111
00M
DI-UFG210
84.42+0.32
74.79+
0.62
85.45+0.18
84.88+0.84
85.30+ 19.50
95.03+0.119
96.06+0.11
71.01+7.28
P
85.0020.36
75.61+0.60
85.59+0.18
84.55+1.57
93.00=0.61
95.03=0.19
96.1440.09
71.21+6.19
PL-UFG2200 85.20.1..22
76.12.1..2
85.59
85.27.1..15
02.50-10.40
94.7710.27
96.05.10.07
OOM
PL-UFG-LFD 85.6411.36 11.59
11.59 16.0811.33
85.36* 1.339
93. 1.30 96.16"...1.1.... 11.08 96.49 11.04 71.611.2
Discussion on the Results, Scalability and Computational Complexity From both Table 3 and it is clear that our proposed model (pL-UFG-LFD and PL-UFG-HFD) produces state-of-the-art learning accuracy compared to various baseline models. For the datasets Pi.e.,Pubmed and Squirrel) which PL-UFG-LFD and PL-UFG-HFD are not the best, one can observe that PL-UFG-LFD and PL-UFG-HFD still have nearly identical learning oulcomes compared to the best pL-UFG results. This suggests even within the PL-UFG with controlled framelet dynamics, by adjusting the values of μ and p, our proposed models are still able to generate stato-of-tho-art learning results with the computational complexity largely reduced comparod to the PL-UFG and UFG. This observation directly verifies the observation in Section 3.3 In addition, due to the reduction of computational cost, our dynamic controlled models (pI-UFG-ILFD and pL-UFG-HFDD
16

---

Published in Transactions on Machine Learning Research (02/2024)
Table 4: Test accuracy (%) on heterophilic graphs. the top learning accuracy is highlighted in bold and the second accuracy underlined.
Method
Chameleon
Squirrel
Actor
Wisconsin
Texas
Cornell
ML.P
48.82+1.43
34.30+1.13
41.66++0.83
93.45+2.09
71.25+12.99
83.33+4.55
GCN
33.71=2.27
26.19=1.34
33.46#1.42
67.9018.16
53.44=11.23
55
15.68=10.57
SGC
33.83.1 .69
26.8911.98
32.08.12.22
59.56111.19
64.3817.53
43.18111641
GAT
41.95+2.65
25.66+1.72
33.64+3.445
60.65+11.08
50.63+28.36
34.09+-29.15
JKNet
33.50=3.40 26.95+1.29
31.14+3.63
60.4218.70
63.75
45.45+99.99
APPNP
34.61+3.15 32.61+0.93
39.11+1.111
82.41+2.17
80.00+5.38
60.98+
GPRGNN
34.2314.00 34.01
34.6310.58
86.11
84.38-11.20
66.29
UFG
50.11+1.67
31.48:2.055
10.13:1.111
93.52:2..36
84.69
83.71
PGNN10
19.04+1.16
34.79+11001
40.91+1.41
94.35++216
82.00=11.31
82.73+6.92
PGNN15
19.12
34.86=1.225
40.87+1.47
94.72:11.11
81.50110.70
10.97
PGNN20
19.34=1.15
34.97+1.41
40.83+1.81
94.44+1.75
84.38=11.52
11.06=10.18
PGNN2.5
19.16.1.40
3..9411.57
40.7811.51
94.3511.166
83.38
81.8218.86
PL-UFG1100
56.81+1.69
38.81
41.2611.66
96.4810.94
86.13
86.0613.16
PI-UFG14.
56.89+1.17
39.73+1.22
10.95
96.4811.077
87.00
86.52-12.29
PL-UFG12.0 2.0
56.24+1.08 .22
39.72+1.86
40.95
96.59+0.72
86.50+8.84
85.30+2.38
PL-UFG12.5
56.11 .55
39.38=1.78
41.0410.99
95.3411.64
89.00-4.99
83.9413.58
PI-UFG21.
55.51 .53
36.9411..69
19.25
93.98:12.91
85.00-5.27
87.73-1-2.49
PL-UFG21.5
57.22+1.19
39.80+1.42
40.89+0.75
96.48+0.94
87.63+5.32
86.82+11.67
PL-UFG22.0
56.1910.99
39.74+1.66
41.01
96.1411.16
86.5018.84
85.3012.35
PL-UFG2225
15.69-11.15
39.30+1.668
40.86:10.74
95.80+1.44
86.38
84.35
PL-IUFG10
55.8011..98
38.4311.22 1.26
32.84116.54
93.9813.47
86.2516.89
8..27
PL-IUFG15
55.65=11.90 .96
38.40+1.52
41.00:0.99
96.48+1.29
87.2543.61
86.21
PL-IUFG20
55.95 .99
38.33
41.25:10.44
96.25.1.225
88.7544.97
83.91-13.78
PL-1UFG25
55.56+1.60
38.39+1.48
40.55++0.50
95.28+2.24
88.50+7.37
83.64+3.88
PL-UFG-HFD 58.60 1174
39.63+2.01
44.75 96.64-1.77 89.31+8.40 $8.97"+3.36
show strong capability of handling the large-scale graph dataset, which is challenging issue (scalability) for some GNNs especially multi-scale graph convolutions such framelets (Zheng et al. 2022b) without additional data pre-processing steps. Accordingly, basclines on Arxiv datascts. Lastly, one can also find that most of the improvements between the learning accuracy produced from our model and the baselines are significant.
check that PL-UFG-LFD outperforms all included
4.4 Limitation of the Proposed Models and Future Studies
First, we note that our analysis the convergence, energy dynamic, and equivalence between our proposed model can be applied or partially appliod to most existing GNNs. Based on what we have claimed regarding the theoretical perspective of pL-UFG, although we assessed model property via different perspectives, eventually, all theoretical conclusions come to the same conclusion (i.e., the asymptotic behavior of pL-UFG).
17

---

Published in Transactions on Machine Learning Research (02/2024)
Therefore, it would be beneficial to deploy analyzing framework to other famous GNNs. Since the main purpose of this paper is to re-assess the property of pI.-UFG, we leave this to future work.
In addition, to induce LFD/HFF to pL-UFG, we set the value of 0 as constant according to Proposition 3 However, due to the large variety of real-world graphs, it is challenging to determine the most suitable 0 when we fix it as a constant. This suggests the exploration of controlling the model's dynamic via selecting 0 is still rough. Moreover, based on Definition [ the homophily index of graph summary statistic over all nodes. However, even in the highly homophilic graph, there are still some nodes with their neighbours with different labels. This suggests the index is only capable of presenting the global rather than local labelling information of the graph. Accordingly, assigning constant 0 to induce LFD/HFD might not be able to equip PL- with enough power to capture detailed labelling information of the graph. Therefore, another future research direction is to potentially explore the design of 8 via the local labelling information of the graph. Finally, we note that another consequence of setting 0o,1 and 01,1 as constant such setting narrows the model's parameter space, as one can check the only learnable matrix left via explicit part of pI.-UFG (eq- (9) is W. Accordingly, the narrowed parameter space might make the solution of the model optimization apart from the desired solution as before, causing potential increases in learning variance.
5 Concluding Remarks
In this work, we performed theoretical analysis on pL-UFG. Specifically, we verified that by choosing suitable quantities of the model parameters (μ and p), the implicit propagation induced from p-Laplacian is capable of amplifying or shrinking the Dirichlet energy of the node features produced from the framelet. Consequently,, such manipulation of the energy results in stronger energy dynamic of graph framelets and, therefore, enhances the model's adaption power on both homophilic and heterophilic graphs. We further explicitly showed the proof of the convergence of PL-UFG, which, to the best of our knowledge, fills the knowledge gap, at least in the field of p-Laplacian-based multi-scale GNNs. Morcover, we showed the equivalence between PL-UFG and the non-linear graph diffusion, indicating that pL-UFG can be trained via various training schemes. Finally, it should be noted that for the simplicity of the analysis, we have made several assumptions and only focus on the Haar type frames. suffices in regard to the scope of this work. However, it would be interesting to consider more complex energy dynamics by reasonably dropping some of the assumptions or from othe types of frames, we leave this to future work.
References
Wendong Bi, Lun Du, Qiang Fu, Yanlin Wang, Shi Han, and Dongmei Zhang. Make heterophily graphs better fit GNN: A graph rewiring approach. arXiv:2209.08200 2022.
Cristian Bodnar, Francesco Di Giovanni, Benjamin Paul Chamberlain, Pictro Liò, and Michael M Bronstein. Neural Sheaf diffusion: A topological perspective on heterophily and oversmoothing in GNNs. Advances in Neural Information Processing Systems, 35, 2022.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Gcometric deep learning: grids, groups, graphs, geodesics, and gauges. 0rX4::210,.18/78, 2021.
Ben Chamberlain, James Rowbottom, Maria I Gorinova, Michael Bronstein, Stefan Webb, and Emannele Rossi. GRAND: Graph neural diffusion. In Proceedings of the International Conference on Machine Learning, pp. 1407-1418. PMTRR 2021a.
Benjamin Chamberlain, James Rowbottom, Davide Eynard, Francesco Di Giovanni, Xiaowen Dong, and Michael Bronstein. Beltrami flow and neural diffusion on graphs. Advances in Neural Information Processing Systems, 34:1594-1600 2021b.
Jialin Chen, Yuelin Wang, Cristian Bodnar, Rex Ying, Pictro Lio, and Yu Guang Wang. Dirichlet energy enhancement of graph neural networks by framelet augmentation. arxv:22311.06 1.05767, 2023.
18

---

Published in Transactions on Machine Learning Research (02/2024)
Qi Chen, Yifei Wang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Optimization-inuuee graph implicit nonlinear diffusion. In Proceedings of the International Conference on Machine Learning, pp. 3648 3661. PMLR, 2022.
Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In Proceedings of the International Conference on Learning Representations, 2021.
Fan RK Chung. Spectral graph theory, volume 92. American Mathematical Soc., 1997.
Francesco Di Giovanni, James Rowbottom, Benjamin Paul Chamberlain, Thomas Markovich, and Michael M Bronstein. Understanding convolution on graphs via energies. Transactions on Machine Learning Research, 2023.
Bin Dong, Sparse representation on graphs by tight wavelet frames and applications. Applied and Computa- tional Harmonic Analysis, 42(3):452-415 2017. doi: att.:...cca..215.000
Pavel Drábek and Stanislav I Pohozaev. Positive solutions for the p-Laplacian: application of the fibrering method. Proceedings of the Royal Sociely of Edinhurgh Section A: Mathematics, 127(4))703777 726, 1997.
Guoji Fu, Peilin Zhao, and Yatao Bian. p-Taplacian based graph neural networks. In Proceedings of the
International Conference on Machine Learning, pp. 6878 6917. PMLR, 2022.
JP García Azorero and 1 Peral Alonso. Existence and nonuniqeeness for the p-Laplacian. Communications in Partial Differential Equalions, 12(12):126 202, 1987.
Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In Proceedings of the International Conference on Learning Representations, 2019.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning large graphs. Advances in Newral Information Processing Systems, 30, 2017.
Andi Han, Dai Shi, Zhiqi Shao, and Junbin Gao. Generalized energy and gradient flow via graph framelets. 0tt.2210.0412,, 2022.
Andi Han, Dai Shi, Lequan Lin, and Junbin Gao. From continuous dynamics graph neural networks: Neural diffusion and beyond. rrXiv:2310.1012/ 2023.
Bernd Kawohl and Jiri Horak. On the geometry of the p-Laplacian operator. Discrete and Continuous Dynamicul Systems S, 10(4):799 813, 2017.
Thomas N Kipl and Max Welling, Semi-supervised classification with graph convolutional networks. Proceed- ings of the International Conference on Learning Representations, 2016.
Remigijus Paulavičius and Julius Žilinskas. Analysis of different norms and corresponding lipschitz constants for global optimization. Technological and Economic Development of Economy, 12(4):301-366; 2006.
Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-GCN: Geometric graph convolutional networks. In Proceedings of the International Conference on Learning Representations, 2019.
Zhiqi Shao, Dai Shi, Andi Han, Andrey Vasnev, Yi Guo, and Junbin Gao. Enhancing framelet GCNs with generalized p-Laplacian regularization. International Journal of Machine Learning and Cybernetics, pp- 1 1-21, 2023. URL https://link... springer.....e.erre
Dai Shi, Yi Guo, Zhiqi Shao, and Junbin Gao. How curvature enhance the adaptation power of framelet GCNs. arXix:2807.0978 2023a.
Dai Shi, Zhiqi Shao, Yi Guo, and Junbin Gao. Frameless graph knowledge distillation. orXiv:2307.0663 2023b.
19

---

Published in Transactions on Machine Learning Research (02/2024)
Dai Shi, Andi Han, Lequan Lin, Yi Guo, Zhiyong Wang, and Junbin Gao. Design your universe: A physics-informed agnostic method for enhancing graph neural net arXiv:2401.14588 2024.
Matthew Thorpe, Tan Minh Nguyen, Hedi Xia, Thomas Strohmer, Andrea Bertozzi, Stanley Osher, and Bao Wang. GRAND++: Graph neural diffusion with a source term. In Proceedings of the International Conference on Learning Representations, 2022. URL
net/forum?id-EMau dzvJk.
César Torres. Boundary value problem with fractional p-Laplacian operator. Advances in Noulinear Analysis, 5(2): 146, 2016.
Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liô, and Yoshua Bengio. Graph attention networks. In Proceedings of the International Conference on Learning Representations, 2018.
Yifei Wang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Dissecting the diffusion process in linear graph convolutional networks. Advances in Nearal Information Processing Systems, 34:5758 5769, 2021.
Quanmin Wei, Jinyan Wang, Jun Hu, Xianxian Li, and Tong Yi. OGT: Optimize graph then training GNNs for node classificaaton.. Newral Computing and Applications, 34(24):22209 22222, 2022. URL
http:://aai..rg/111100/////////
Felix Wu, Tianyi Zhang, Amauri Holanda de Souza, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. Simplifying graph convolutional networks. In Proceedings of the International Conference on Machine Learning, pp. 6861-6871. PMLR, 2019.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEFE Transactions on Neural Networks and Learning Systems, 32(1): 4 24, 2020.
Keyulu Xu, Chenglao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi,, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In Proceedings of the International Conference on Machine Learning, 2018.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In Proceedings of the International Conference on Learning Representations, 2019. URL htttps://openreviet
net/foorum?d::ryG66...
Mengxi Yang, Xuebin Zheng, Jie Yin, and Junbin Gao. Quasi-framelets: Another improvement to graph neural networks. 2xi::2201.04722 2022.
Xin Zheng, Yixin Liu, Shirui Pan, Miao Zhang, Di Jin, and Philip S Yu. Graph neural networks for graphs with heterophily: A survey, arXiv:2202.0m0 07082, 2022a.
Xuebin Zheng, Bingxin Zhou, Junbin Gao, Yuguang Wang, Pietro Lió, Ming Li, and Guido Montufar. How framelets enhance graph neural networks. In Proceedings of the International Conference on Machine Learning, pp. 12761-12771. PMLR, 2021.
Xuebin Zheng, Bingxin Zhou, Yu Guang Wang, and Xiaosheng Zhuang. Decimated framelet system on graphs and fast g-framelet transforms. Journal of Machine Learning Research, 23:18-1, 2022b.
Bingxin Zhou, Ruikun Li, Xuebin Zheng, Yu Guang Wang, and Junbin Gao. Graph denoising with framelet. regularizer. Xrx:::211. .03264, 2021.
Dengyong Zhou and Bernhard Schölkopf. Regularization on discrete spaces. In Joint Pallern Recognition Symposium, pp. 361-368. Springer, 2005.
Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems,
2020.
Meiqi Zhu, Xiao Wang, Chuan Shi, Houye Ji, and Peng Cui. Interpreting and unifying graph neural networks with an optimization framework. In Proceedings of the Web Conference 2021, pp. 1215-1226, 2021.
20

---

Published in Transactions on Machine Learning Research (02/2024)
A Appendix
A.1 Detailed Formulations
In this section, we provide detailed formulation as the preliminarics of the graph p-Laplacian operator shown in Definition
A.1.1 About p-Laplace Operator
Definition 5 (The p-Laplace Operator Drábek & Pohozaev 1997) ). Let Ω C Rd be a domain and u is a
function defined on Q. The p-Laplace operator over functions defined as
Au = V. ((Vluu]'=>Vvu)
where V is the gradient operalor and -| is the Euclidean norm and is a scalar salisfying < < +00. The p- p-Loplace operator is known as a quasi-linean elliptic partial differential operator.
There is a line of research on the properties of p-Laplacian in regard its uniqueness and existence Garria Azorero & Peral Alonso 1987), the geometrical property (Kawohl & Horak 2017), and boundary conditions on so-called p-Laplacian equalion (Trres 2016).
The concept of p-Laplace operator can be extended for discrete domains such as graph (nodes) based on the concepts of the so-called graph gradient and divergence, see below. One the recent works (Fu et al., 2022) considers assigning an adjustable p-Laplacian regularizer to the (discrete) graph regularization problem that is conventionally treated as a way of producing GNN outcomes (i.e., Laplacian smoothing) (Zhou & Schölkopf] 2005). In view of the fact that the classic graph Laplacian regularizer measures the graph signal energy along edges under L2 metric, it would be beneficial if the GNN training process could be regularized under Tpp metric in order to adapt to different graph inputs. Following these pioneer works, [Shao et al. (2023) further integrated graph framelet and a generalized p-Laplacian regularizer to develop the so-called generalized p-Laplacian regularized framelet model. It involves a regularization problem over the energy quadratic form induced from the graph p- p-Laplacian. To show this, we start by defining graph gradient as follows:
To introduce graph gradient and divergence, we define the following notation:
Given graph G = E W), let Fv = [F|F: V + Rd) be the space of the vector-valued functions defined on V and F& = {gig E + Rd be the voctor-valued function space on edges, respectively.
Definition 6 (Graph Gradient (Zhou & Schölkopf 2005 ) For a given function F E Fv, its graph gradient is an operator :w:Fy + Fe defined as for all (Vi, U2) E
WiJ
WL di.i
F) =
Eg
f.r
(30)
d3s
where f, and Eg are the signal vectors on nodes v, and Uy, i.e., the rows of F.
For simplicity, we denote Vw F as VF as the graph gradient. The definition of the (discrete) graph gradient is analogized from the notion of gradient from the continuous space. Similarly, can further define the so-called graph divergence:
Definition 7 (Graph Divergence Zhou & Schölkopl 2005 ) The graph divergence is an operalor div Fe + Fy which is defined in the following way. For a given function g E Fe, the resulting div(g) E Fv satisfies the following condition, for any functions F E Fv,
(VF,g) = (F, (F,-Fiidf ddo(g))
(31)
It is easy to check that the graph divergence can be computed by:
Wijj
div(g) =
(gli. -g]. f))
(32)
d
21

---

Published in Transactions on Machine Learning Research (02/2024)
With the formulation of graph gradient and divergence, we can define the graph p-Laplacian operator and the corresponding p-Dirichlet form (Zhou & Schölkopf 2005
Fu et al. 2022 as illustrated in Definition 3
A.1.2 About LFD and HFD Dynamic
We provide some detailed formulation on the so-called Low frequency dominance (LFD) and high frequency dominance (HFD) of graph framelet. In the work (Di Giovanni et al. 2023), the propagation of GNNs was considered as the gradient flow of the Dirichlet energy that can be formulated as:
N N
E(F)
ΣΣ
IV
WiJ E d3s
Wi fg de.s
(33)
2 1=1
dass
and similarly by setting the power from 2 to p, we recover the p-Dirichlet form presented in eg.
12 The
gradient flow of the Dirichlet energy yields the so-called graph heat equation (Chung) 1997
as F(k)
-VE(F())) (L)(), Its Euler discretization leads to the propagation of linear GCN models (Wu al.
2019: Wang et al. 2021 Following this observation, the work (Han et al.. 2022 Di Giovanni et al. 2023
Shi et al.) 2023a) also shows even with the help of the non-linear activation function and the weight matrix via classic GCN (eq. [T), the process described is still dominated by the low frequency (small Laplacian cigenvalues) of the graph, hence eventually converging to the kernel of L, for almost every initialization. To quantify such behavior, Di Giovanni et al. (2023) and Han et al. (2022) considered a general dynamic as (bk) = oNN, (F((), ,k), with GNNo(.) as an arbitrary graph neural network function, and also characterizes its behavior by low/ (high-frrquency dominance (L/HFD).
Definition 8 (Di Giovanni al. 2023)). (kk CNNN(F((), is Low- Now--requency-Doee Dominnn (LFDD) if 00, and is High- Frequeeyr Dominant (HFD) if
B(F()) (k)]) + 0 0000 C0.
- pi.
Lemma 3 (Di Giovanni et al. 2023 ). GNN model LFD (resp. HFD) if and only if for each t, + 00, there crists a sub-sequence indered by ky; - and F such that F(k5))/|[F(() + Fzo and LF = 0
(resp. LF = Poo)
To employ the similar notions of LFD/HFD, Han et al. 2022 developed framelet Dirichlet energy and analyzed the energy behavior of both spectral (eq. (9h) and spatial framelet (eq. (10) convolutions. Specifically, let
E (F) Tr (T((W,,,F)) W..FF.,.) (W....)).........) Tr (( (i,
for all (r, () I. The generated framelet energy given by:
E EF*(F)
E (F) + Σ
E (F)
r,
1 Σ vec(F)), (R @ W.T W & di )ee(()))))))))))))))))))
(34)
where the superscript wFrm stands for the framclet convolution.
This definition is based on the fact that the
total Dirichlet energy is conserved under framelet decomposition (Han et al. 2022 Di Giovanni et al. 2023). By analyzing the gradient flow of the framelet energy 3 defined above, Han et al. (202)) concluded the energy dynamic of framelet as:
Proposition 3 ((Han et al. 2022 ). The spectral graph framelet convolution eq. g with Haar-type filter (i.e. R R= 1 in the case of scaling function set) can induce both LFD and HFD dynamics. Specifically, let 0o. 1N and 0, = f1l for r ===1,.. 1,...,,,, L,e.,....JJ where IN is size N vector of all 1s. When 0 E (0,1), the spectral framelet convolution is LFD, and when > 1, the spectral framelet concolution is HFD.
Remark 7. Proposition 3 suggests that graph framelet can naturally induce both two types of dynamic without further modification, and this supports our claim on the property of graph framelet in Section 3.3
"Similar to the requirement on our p-Laplacian based framelet energy (EPP (F((+1))) to thoroughly verify the framelet energy in eq. 334 is type of energy, we shall further require: V'B (IF) = Pr,t WT, Wee W & W Lw,, is symmetric, which can be satisfied by requiring both Ω and W are symmetric.
W.T.We
22

---

Published in Transactions on Machine Learning Research (02/2024)
A.2 Formal Proofs
A.2.1 Detailed Proof of Proposition 1
Theorem 2 (Repeat of Theorem 1 Given graph g(V, &, W) with node features X, if (h),, g(h), M and F(N) are updated according to eq. 18). then there crist some real positive value μe which depends on the inpul graph (G, X) and the quanlity of updaled in each ileration, such that:
(Fltttl) s ()(F())),
where (F) == Sp (F) # Y
Proof. First, write:
M
Wis 2
p-2
Td'[I (wiil))
d'(I ( (w))11p)
(35)
F
FW (w:))1
The derivative of the regularization problem is defined in eq.
15 is:
(F)
FF
(2) YY,)+ Σ
M
(1)
1
Vw F ([i
OF 5,:
8,:
v daiway
F(2)
uy
-F (K) Y..) Σ
M(a)
(((h)
(K)
dii
4i
v disd
3.t
By 22l
(36)
M (K)
=(2u + Σ
M /d ((
(K)
My v diedi
My myn
M!
(k)
g(k) i
Y.
+ a
(k)
Σ
(k)
(k)
2y v digdis
0
Oi
Thus, according the update rule of (k+1) in eq.
18 we have
(F)
F
(R++))
6,
(37)
OF
(k)
(P())
O
For our purpose, denote the partial derivative at F(+) of the objective function with respect to the node feature F., as
(F)
(C
(38)
oF.:
F(-)
23

---

Published in Transactions on Machine Learning Research (02/2024)
For all i.j E [M] let R liee be disturbance acting on node i. Define the following:
NOO Wis
Wi p(.
Wip pll) V Djs
|V
Dia
3:
N Wi,
Wis
(F
(2) +v) V Djs d
W
(())
Di.s
M
(K)
N! Se (F(K)).
(K)
MD (1k)
-Noh Sa.5 (F +v)
(39)
M
(() 1/ au
B 2pa ((K)
+ 2p
Di.
(
F((+1) (())
3: + B Yio.
Dr..D,.
where Sez (F) is defined eq.
19 and (()) + v means that only applies to the i-th of (
Similar to eq. 37 we compute
(2) (F t.
+v)=
(IF(H) + +v) - F(.ttt)
(40)
(k)
ais
Hence from both eq.
37 and eq.
40 we will have
| (k)
+ - (00)-a0-22 (1) ||| =
(FF i.:
+v) - -((++1))
(F(
p
((tt)))
[k)
(6)
2,
aii
|v|
+
1
[(()] - 1((+1)
1
F(())
(k)
F (k+1))
(K)
((k)
(b)
6,1
1
|v|
+
1
F
(k)
1
F
1k++1)
F (k+1)
(k)
(k)
(0))
(k)
ai,i
a2
ai
(((A)
M
(k)
(R)
M.
1
|v||
+
Σ
F
(k)
Σ
F((h)
(DD.D
(k)
Di.
D2.5
Di.,Dis
a
M
M((A)
MSA
241
||v|+
Σ
liv|l
Di,3
Di,
M.(K)
M(A))
M.
M.(
Σ
ij
F
Σ
(k)
DeS
Plywey
Note that in eq. (39 1|-1 is the matrix L2 norm raised to power p p-2, that is ||||-- = ( ( a )
It is known that the matrix Ly norm as a function is Lipschitz (Paulavičius & Zilinskas 2006 so is its
exponential to p 2-2. 2. Furthermore, it is easy to verify that J|N' N $ c|lv| due to the property of N and the following holds
N'. Hence, according to Lemma [
M 1 NOHh < [[]l
With slightly abuse of notation, we denote N' as the matrix after assigning the disturbance the matrix
24

---

Published in Transactions on Machine Learning Research (02/2024)
Combining all the above, we have
M
(k)
(F +v)
- +v)
Σ
+ +2 + o19, olg, v. v. ||vl
(41)
Dia
where ogG, v, p) is bounded. It is worth noting that the quantity of o(9, v, x, is bounded by
M 1. M (A)
M
M
My
(R) Vis.D
M,
||v|+ +
Σ Bymea
F
Σ
Di
Dio
My
Let 0= oG. v, x)), [N) and E NR..
By the Taylor expansion theorem, we have:
(F (k) +mm 2 (F + 7
(F (b) temm.). n..dee Vi
= - + (0C m +
(ocg (F!.) + +emm. -ac% (F!!i) mhi de
00 (
m
Jacs (F( + exm.: - OC2 (F(()) |m de
s c + ( mmm+
+
+o n 1n
(K]
where the last inequality comes from eq- 41
Taking % a (k) and mi, = a ( in the above incquality gives
2g (p(h)) as Oc (p(
(K)
ai
(k)
)C ( 112
1
1
(k)
+
a
2(k)
|a
(F
ai,i
= (F(())
2 a (1
1 (M)
(K)
| F
(42)
Given that T is bounded, if we choose large μ. e.g., 2p > o, we will have
(k)
0.
M D..
E Eeyve
+ 2p
Thus the second term in eq. 42 is positive. Hence we have
(F 1.
(1+
C (F( a ac < (F(
(k)
This completes the proof.
A.2.2 Detailed Proof of Proposition 2
Proposition (Repeat of Proposition 2). Assume is connected, unweighted, and undirected. There exists sufficiently large value of μ or small value of p such that E (F) will stay away above at each iterative layer l and increases with the increase of μ or the decrease of p.
Proof. We start with the definition of the generalized Dirichlet energy above, we can re-write (PF((((++1)) in the following inner product between vee(F(k+1)) and vec(F(0)), based M, a, and the iterative scheme
25

---

Published in Transactions on Machine Learning Research (02/2024)
defined in eq.
18p:
E PF (F(k+1)) (vec(F((+1)).
Le 8 1I Lo @ (a 1112 D-12)) vec(F(At1)) + (Io @ 2ua ")e
2 (voc(E((+1)), voo(F((+1))) (vo((B(
(coo(F(*+1)) 2 I, @ (a (a D 112) voo(F((t1))
-Ie @ 2pa (k+ (vecc(())FFFF))))
(43)
Based on the form of eq- (3)) is straightforward to see that to let EPF (F((tt))))) > 0 and further increase with the desired quantities of μ and p. it is sufficient to require
Ie @ (a (a 1 D vec(F((+)))- - (Ie @ 2 (vo < 0.
(44)
To explicitly show how the quantitics of and affect the term in eq.
44 we start with the case when
k == 0. When k = 0, eq.
44
becomes:
L @ (a D 1/2 D-1/2 (1) D 1/2) vec(F(()) - (I. @ 2uaa (2) vec(()l)))
-Ie @ (a D-1/ M/2 (1) D 1/3) vec (a D-12
M (0)
1/2 (0) + 2ua (0(00) (I. @ 2pa
2ua (1)
-I. @
(a DC ( (L. 8 (a D-
1/2 (0 + 2ua (0) vec(F(())) (Ie @ 2pa )vec(((()))
-L @
II a D
1/2 M
1/2 ( -2ua(1) vec((F(o).
(45)
We note that, in eq. 45), the i-th element of (II'o - 10 11) 2pa can be computed as:
II ==0
oo dil
1/2 M
1 + (a 2 / - 2ua!!
(0)
)
1/
di,s
+ 2p
Vdridgss
+
0=
Uyruu;
VW
((.,)))1
Mi3
(0)
+ 2/z
2/1/
Σ
+ 2/1
di,;
di,i
Nyyves
Byrwes
(11)
2p/
Σ
3.j.
+ 2p
di,s
1--
V VwF ((;,5)))1 p-2
+
M Eoyoo
(
M(21)
+ 2p do,.
V dendes
+ 2/1
v di
thit
V (() (1) p-2
M((1)
1)
2u/
Σ
+ 2/4
2pf/
Σ
241
(46)
M Enywes de + 24)
di.i
dis
Ny
Sttretly speaking, one shall further require all elements in F(++1) larger than equal As this can achleved by assigning non-linear activation function (L.e. ReL.U) the framelet, omit here in our main analysis.
26

---

Published in Transactions on Machine Learning Research (02/2024)
Now see that by assigning sufficient large of μ small value of p, we terms
like
in eq. (46) are getting smaller. Additionally, we have both
12 +2p
2p/
Enjmer
E
Li + 24 and 2/// Enjore + 2/1 Therefore, the summation result of eq. (46)
de,
di
tends to be negative. Based on eq.
43
E
(F(k+1) will stay above 0.
For the case that h 2 1, by taking into the iterative algorithm eq.
18 eq.
45
becomes:
& (TT 1/2 1 1/2
k+1
k+1
(2 - ((a((+t)) )) voc(F
-
D-122
II
lh 1/2 M
1/2 D-1/3
==0
M=k-s
Applying the same reasoning as before, one can verify that both II ( D-12 1/2 Est (I - 1/211112 (2 approach to zero due to their forms defined in eq. (17)
and
and (k+1) goes to 1. Therefore, the above equation tends be negative, yielding a positive EPP (F(K+I)) and that completes the proof.
A.2.3 Detailed Proof of Lemma 2
Lemma 4 (Repeat of Lemma Assuming & is connected, the forward Euler scheme presented in eq. 28 admits a generalized non-linear diffrusion on the graph. Specifically, we have:
1/2-111
F(() B =a aa (div(/VEW))p-2ppu) + (*)(( (k)
(47)
Proof. The proof can be done by verification. We can explicitly write out the computation on the i-th row of the L.H.S of the above equation becomes: First let us denote the rows of F(k) as (i)'s.)..
Σ
(k) du 1/2
M dij
(K) 1/2)
(() (()) + Y(1)
Mij
(())
1
(1)
+ 2ucce (4) p(()) (i)
-6,
Vihi V
(2)
Cymes
ais
=0,
Σ Bymmus
Mij.
((-)))))
Σ
Mij dis
+ 244 p(A)(i)
+ 2pa
(k) (0) (i)
Vduy das
=0,
Wijj d..i
Fwi
(V
(Ww ((k)
Vdj
V
Wii p(()) di,
+ 2/1 Σ p(())
Fly mo
My mas
(k) ((()
2ua
0, ((), + 2pa (k) d=s + 2ua (x)(2))) (.))
(k)
(48)
When takes from to N, gives the result of the R.H.S of the Lemma according to eq. Thus we complete the proof.
26 and eq.
27
27


Reviews:
[{"review_id": "23ZazlRFVQ", "reviewer": "Reviewer_ucm5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The manuscript delivers a robust theoretical foundation for the pL-UFG framework, making a critical contribution to the comprehension of its intrinsic properties and behaviors—a step that significantly fortifies the theoretical landscape of GNNs.\n\n2. The innovative linkage drawn between theoretical underpinnings and non-linear diffusion processes not only solidifies theoretical concepts but also offers valuable insights for the practical deployment and interpretation of GNNs in real-world scenarios.\n\nWeaknesses:\n\n1. Theorem 1 only shows the non-increasing nature of the objective function related to Problem (18), but fails to establish the convergence of the generated sequence and the convergence rate.\n\n2. While Proposition 2 sheds light on the energy dynamics within pL-UFG, the broader significance and practical ramifications of this analysis remain underexplored.\n\n3. The preliminaries in Section 2 are overly extensive and could be condensed, as they currently delve into established results that would be better suited to a brief summary with appropriate references for further reading.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a theoretical analysis of the graph p-Laplacian regularized framelet network (pL-UFG) and its convergence properties, energy dynamics, and connection to non-linear diffusion processes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Expand on Theorem 1 by providing a more comprehensive analysis of the sequence convergence and establishing the convergence rate.\n\n2. Elaborate on the practical significance of Proposition 2's energy behavior results. How does this behavior influence the model in application, and what can practitioners expect in terms of model performance?\n\n3. Shorten the preliminaries section by focusing on the essentials and referencing existing work for further details.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "j2a1uv54Ce", "reviewer": "Reviewer_X4Tw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper provides a detailed analysis of an iterative algorithm's convergence and energetic behavior for solving the implicit layer formulation. The incorporation of a message-passing scheme adds a layer of novelty to the framework, setting it apart from existing approaches.\n- The paper has the potential to serve as a valuable reference for researchers in the domain, contributing to the advancement of knowledge in this specialized area.\n- The authors has diligently revised the paper, taking into account feedback from reviewers to clarify the mathematical formulations and parameters. They have contextualized the motivation for their work more effectively and complied with requirements for providing detailed hyperparameter settings and conducting statistical significance testing in the experimental section.\n\nWeaknesses:\n- The structure of the paper remains somewhat disorganized, hindering readability. Certain proofs and mathematical derivations come across as overly verbose and symbol-heavy, detracting from the clarity of the presentation. The extensive focus on formulaic derivations occupies a significant portion of the paper, which might overwhelm the narrative flow and impede the understanding of the core contributions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper explores \"Generalized p-Laplacian Regularized Framelet Graph Convolutional Networks\" (pL-UFG), targeting the challenge of applying Graph Neural Networks (GNNs) to heterophilic graph structures. The model employs an implicit layer designed to regulate Dirichlet energy across node features, addressing the issue of over-smoothing often observed in GNNs.\n\nThe paper provides evidence of convergence for the model's iterative process when an appropriate regularization parameter is selected. They also discuss the capacity to adjust model parameters to manage the Dirichlet energy of node embeddings, potentially preventing the loss of signal detail. \n\nThe paper proposes a method for adjusting the frequency of filters in the model, which could allow for adaptability to both homophilic and heterophilic graph data. These theoretical considerations are accompanied by empirical experiments that aim to validate the proposed concepts on datasets representing both homophilic and heterophilic graphs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "To enhance the paper and improve its readability, I recommend the following revisions:\n- Streamline Mathematical Derivations: The paper would benefit from a more succinct approach to the presentation of mathematical content. Lengthy and complex derivations should be condensed, with the main text focusing on the essential steps that are crucial to understanding the results. Additional and intermediate steps could be moved to an appendix, ensuring that the core narrative is not obscured by detailed calculations.\n- Reorganize Content for Coherence: The paper requires careful attention to content organization to avoid redundancy and ensure that each section contributes meaningfully to the overall narrative. Moreover, it has been observed that tables have been misplaced within the reference section, which disrupts the document's flow and could cause confusion.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZS6pz49oIf", "reviewer": "Reviewer_3TyJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I list the strengths and weaknesses below, from the angle of clarity and correctness. \n\nStrength:\n- the topic under consideration should be of interest to a sizable audience of TMLR. \n- the number of baselines and datasets used in the experiments are comprehensive. \n- the table of notations helps a bit with clarity \n\nWeaknesses: \n\n1. Many definitions are unclear/stated in sloppy manners. \n\n- In definition 1 of homophily and heterophily, a definition based on the criterion H(G) -> 1 or H(G) -> 0 are given. However, 1. which parameter is being sent to infinity is unspecified 2. the definition implies that a *sequence* of graphs satisfies heterophily or homophily, rather than a single graph G. \n\n- in (17), it is said that phi is positive and convex. But then in the examples stated immediately after, the authors picked phi(x) = x^p, without specifying the domain or the choice of p. Clearly, for p odd and/or for x negative, the function can be neither positive nor convex. \n\n- in proposition 1 and below, the derivative of phi is used and stated. However, just from the positive and convex conditions alone, phi is not guaranteed to be differentiable everywhere (there could be a countable set where it is not differentiable). The authors should state what happens in those scenario, or add extra conditions on phi to make sure the derivative is rigorously justified. \n\n- the authors say \"iterative algorithm induced from the p-Laplacian implicit layer admits a generalized non-linear diffusion\nprocess\". They also say on page 18 \"Furthermore, if we treat the iteration equation eq. (19) as a diffusion process\"...however, it is unclear what they mean, rigorously, by a diffusion process, since they never formally defined it. A diffusion process is a continuous time markov process, but the schemes presented are discrete. The authors should clarify this formally. \n\n- I would like to emphasize that while there are many unclear points in the authors' writing and notation, I find that the concepts introduced are quite natural and intuitive in this context. To the expert readers, we can infer what the authors are trying to say/do. And I am not saying that the author's results are incorrect. However, I find the results proposed by the authors, in current form, to be insufficiently rigorous and clear for scientific publication. In its current form, it is difficult to check and verify the correctness of all the claims that the authors make. As suggested by the prior reviewers, the paper is dense and the mathematics is not organized in a sufficiently coherent manner. There are also lot's of results and notions that are directly imported from other papers. \n\n2. Clarity of writing, formatting, and organization: there are many places where the formatting, wording, writing and notations are rushed and unclear. \n\n- In the first two paragraphs in blue, the citations are not consistent (some are cited with parenthesis, some without). \n\n- notations such as \\approx in the proof of lemma 3 is not rigorously defined. I suggest that the author make explicit the approximation or limiting argument that they are making, rather than resort to \\approx notations. \n\nI echo the opinion expressed by the reviewers of the prior iteration of the draft that the calculations of this paper are long, stacked together but not organized in a sufficently readable/coherent manner. I suggest significantly compressing the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors analyze the theoretical properties of the \"graph p-Laplacian regularized framelet network\" (pL-UFG), which is a framework/method originally designed to allow GNNS to fit both homophilic and heterophilic graphs well. \n\nThe main claims/contributions that the authors make include:\n\n1. they provide a theoretical convergence analysis of pL-UFG\n2. they demonstrate that the generalized Dirichlet energy of the graph under consideration through pL-UFG remains nonzero. \n3. they provide a perspective from the angle of non-linear diffusion processes\n4. they propose two new variants of pL-UFG and provide experiments to demonstrate their new methods", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. make sure all the definitions/notions mentioned above rigorously/formally defined in the paper.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The decision among the reviewers is mixed. Two reviewers are in favour of accepting this submission, while one reviewer recommended "leaning reject". The reviewer's main concerns are the lack of a theοretical result that establishes the full sequence's convergence and the lack of analysis on the convergence rate.


Overall, all reviewers and myself have found merit in the paper for publication. However, the reviewers have mainly complained about the paper's disorganized structure and the clarity of presentation. The authors improved the manuscript during the review process based on the reviewers' comments, but I still feel that further improvements need to be made. For instance, in many cases, \cite is used instead of \citep, e.g., "namely homophily and heterophily Zheng et al. (2022a)". Many sentences need to be rephrased/rewritten in a more elegant way, e.g., "With the former most of the nodes are connected with those nodes with identical labels, and the latter is not Pei et al. (2019)". There are also several typos that need to be fixed, e.g., "(pL-UFG-HFD). we" ==> "(pL-UFG-HFD). We".

I am thus recommending accept with a minor revision, and I expect the authors to improve the quality of presentation in the final version of the paper.
Decision: Accept (poster)