DiGRAF: Diffeomorphic Graph-Adaptive Activation Function
Authors: Krishna Sri Ipsit Mantri
Xinzhi Wang
Carola-Bibiane Schönlieb
Bruno Ribeiro
Beatrice Bevilacqua
Moshe Eliasof
Venue: NeurIPS 2024 poster (2024)
Keywords: Graph Neural Networks
Graph Activation Functions
Primary area: graph_neural_networks
TL;DR: We introduce DiGRAF - a novel graph-adaptive activation function for GNNs by learning flexible and efficient diffeomorphisms, and demonstrate its effectiveness on numerous benchmarks and tasks.

Abstract: In this paper, we propose a novel activation function tailored specifically for graph data in Graph Neural Networks (GNNs). Motivated by the need for graph-adaptive and flexible activation functions, we introduce DiGRAF, leveraging Continuous Piecewise-Affine Based (CPAB) transformations, which we augment with an additional GNN to learn a graph-adaptive diffeomorphic activation function in an end-to-end manner. In addition to its graph-adaptivity and flexibility, DiGRAF also possesses properties that are widely recognized as desirable for activation functions, such as differentiability, boundness within the domain, and computational efficiency. 
We conduct an extensive set of experiments across diverse datasets and tasks, demonstrating a consistent and superior performance of DiGRAF compared to traditional and graph-specific activation functions, highlighting its effectiveness as an activation function for GNNs. Our code is available at https://github.com/ipsitmantri/DiGRAF.

DIGRAF: Diffeomorphic Graph-Adaptive Activation Function
Krishna Sri Ipsit Mantri'
Xinzhi (Aurora) Wang" Purdue University vang61710purdue edu
Carola-Bibiane Schönlieb University of Cambridge cbs310cam. ac. uk
Purdue University
mantrikepurdue. edu
Bruno Ribeiro
Beatrice Bevilacqual Purdue University bbevilacqpurruue.c edu
Moshe Eliasof University of Cambridge me5320cam ac
Purdue University
ribeirob@purdue.cm edu
Abstract
In this paper, we propose novel activation function tailored specifically for graph data in Graph Neural Networks (GNNs). Motivated by the need for graph-adaptive and flexible activation functions, we introduce DIGRAF, leveraging Continuous Piecewise-Affine Based (CPAB) transformations, which we augment with additional GNN to learn a graph-adaptive diffeomorphic activation function in an end-to-end manner. In addition to its graph-adaptivity and flexibility, DIGRA AF also possesses properties that are widely recognized as desirable for activation functions, such as differentiabilityy boundness within the domain, and computational efficiency. We conduct an extensive set of experiments across diverse datasets and tasks, demonstrating consistent and superior performance of DIGRAF compared traditional and graph-specifts activation functions, highlighting its effectiveness an activation function for GNNs. Our code is available at https: / /gi thub. com/ Seeiitaatrr/DiGRA.
Introduction
Graph Neural Networks (GNNs) have found application across diverse domains, including social networks, recommendation systems, bioinformatics, and chemical analysis [82, 68). Recent advancements in GNN research have predominantly focused on exploring the design space of key architectural elements, ranging from expressive GNN layers [57, 23, 88, 89, 65], to pooling layers [87, 46, 5, 80], and positional and structural encodings [18, 67, 20]. Despite the exploration of these architectural choices, common trend persists where most GNNs default employing standard activation functions, such ReLU [26], among few others.
Activation functions play crucial role in neural networks, they necessary for modeling non-linear input-output mappings. Importantly, different activation functions exhibit distinct behaviors, and the choice of the activation function can significantly influence the performance of the neural network [60]. It is well- known [61, 72] that, from theoretical point of view, non-convex and highly oscillatory activation functions offer better approximation power. However, due to their strong non-convexity, they amplify optimization challenges [39].
Therefore, middle-grount between practice and theory, it has been suggested that successful activation function should possess the following properties: (1) differentiable everywhere [16. 54]. (2) have non-vanish- ing gradients [16]; (3) be bounded to improve the training stability [48, 16]; (4) be zero-centered to accelerate convergence [16]; and (5) be efficient and not increase the complexity of the neural network [45]. In the context of graph data, the activation function should arguably also be what we define adaptiad, that tailored
"qqual contribution. Equual supervision.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

DIGRAF
(
GNN LAYER
H
T
H
CNN ACT
Figure Illustration of DIGRAF. Node features H and adjacency matrix A are fed GNNEAYER to obtain updated intermediate node features HO) which are passed to our activation function layer. DIGRAF. First, an additional GNN network GNN takes H and as imput to determine the activation function parameters ((), These are used to parameterize the transformation T((), which operates on H produce the activated node features H
to the input graph and capable of capturing the unique properties of graph-structured data, such as degree differences size changes. This adaptivity ensures that the activation function can effectively leverage the structural information present in the graph data, potentially leading to improved performance in graph tasks.
Recent work in graph learning has investigated the impact of activation functions specifically designed for graphs, such as lancu et al. [38] that proposes
max median activation filters, and Zhang al. [92] that introduces GReLU, which learns piecewise linear activation functions with a graph-adaptive mechanism. Despite the potential demonstrated by these approaches, the proposed activation functions still have predefined fixed structures (max and median functions in Iancu al. [38] and piecewise linear in Zhang al. [92].. restricting the flexibility of the activation functions that be learned. Additionally, in the case of GReLU, the learned activation functions inherit drawback points of non diffferentiabilityy which are undesirable according to the properties mentioned above. As a consequence, to the best of our knowledge, none of the existing activation functions prove to be consistently beneficial across different graph datasets and tasks. Therefore, objective is design flexible activation function tailored for graph data, offering consistent performance gains. This activation function should possess many, if all, of the properties recognized as beneficial for activation functions, with an emphasis on blucprint ficxibility, as well as task and input adaptivity.
In this paper, we leverage the success of learning diffcomorphisms, particularly through Continuous Piccewise- Affine Based transformations (CPAB) 124, 25), devise an activation function tailored for graph-structured data. Diffeomorphisms, characterized as bijective, diffcrentiable, and invertible mappings with a differentiable inverse, inherently possess many desirable properties of activation functions, like differentiability, boundedness within the input-oulput domain, and stability to input perturbations. To augment activation function with graph-adaptivity, we employ additional GNN derive the parameters of the learned diffeomorphism. This integration yields our node permutation equivariant activation function, dubbed DIGRAF Diffcomorphismm baaee GRaph Activation Function, illustrated in Figure 1, that dynamically adapts to different graphs. providing flexible framework capable of learning activation functions for specific tasks and datasets in end-to-end manner. This comprehensive set of characteristics positions DIGRA A as promising approach for designing activation functions for GNNs.
Our Approach: DIGRAF.
To evaluate the efficacy of DIGRAF, we conduct an extensive of experiments diverse of datascts across various tasks, including node classification, graph classification, and regression. Our evaluation compares the performance of DIGRAF with three types of baselines: traditional activation functions, activation functions with trainable parameters, and graph activation functions. Our experimental results demonstrate that DIGRAF repeatedly exhibits better downstream performance than other approaches, reflecting the theoretical understanding and rationale underlying design and the properties possesses. Importantly, while existing activation functions offer different behavior in different datasets, DIGRAF maintains consistent performance across diverse experimental evaluations, further highlighting its effectiveness.
The contributions of this work are summarized as follows: (1) We introduce learnable craph-adaptive activation function based on flexible and efficient diffcomorphisms DIGRAF, which we show to have properties advocated in literature; (2) an analysis of such properties, reasoning about the design choices of method; and, (3) comprehensive experimental evaluation of DIGRAF and other activation functions.
Main contributions.

---

ELU
Tush
DIBRAFF
Pironeiae BeLU (K-1)
Passeria Todll [K-1]
Plezenina ReL0 (K=2)
Relld
Piroeiia RaLU (K-5)
Passoria Todl (K-5)
(a) ELU
(b) Tanh
Figure 2: Approximation of traditional activation functions using CPAB and Piecewise ReLU with
varying segment counts K E [1, 2,3) } on a closed interval Ω = [-5, 5), demonstrating the advantage of utilizing CPAB and its flexibility to model various activation functions.
Related Work
Diffeomorphisms in Neural Networks. A bijection mapping function f M -> N. given two differen- tiable manifolds M and N. is termed diffeomorphism if its inverse f :N M is also differentiable. The challenge in learning diffcomorphisms arises from their computational complexity: early research is often based on complicated infinite dimensional spaces [76], and later advancements have turned to Markov Chain Monte Carlo methods, which still suffer from large computational complexity [1, 2. 90]. To address these drawbacks, Freifeld et al. [24, 25] introduced the Continuous Piecewise-Afline Based transformation (CPAB) approach. offering a more pragmatic solution to learning diffeomorphisms by starting from finite- dimee-iimmnnional space, and allowing for exact diffcomorphism computations in the case of ID diffcomorphisms an essential trait in our case, given that activation functions are ID functions. CPAB has linear complexity and is parallelizable which can lead to sub-linear complexity in practice [25]] Originally designed for alignment and regression tasks by learning diffeomorphisms, recent years, CPAB was found to be effective in addressing numerous applications using neural networks, posing as a suitable framework for learning transformation. For instance, Detlefsen et al. [15] learns CPAB transformations to improve the flexibility of spatial transformer layers, Martinez et al. [52] combines CPAB with neural networks for temporal alignment, Weber and Freifeld [81] introduces novel loss function that eliminates the need for CPAB deformation regularization in time-series analysis, and Wang [79] utilizes CPAB model complex spatial transformation for image animation and motion modeling.
General-Purpert Activation Functions. In the last decades, the design of activation functions has seen extensive exploration, resulting in the introduction of numerous high-performing approaches, as summarized Dubey et al. [16]. Kunc and Kléma [45]. The focus
gradually shifted from traditional, static activation functions such as ReLU [26]. Sigmoid [45]. Tanh [35]. and ELU [11]. to learnable functions. In the landscape of learnable activation functions. the Maxout [29] unit selects the maximum output from learnable linear functions, and UU [33] extends ReI. by lcarning negative slope. Additionally, the Swish function [66] augments the SiLU function [19], iiiihtt-eiighted linear unit, with learnable parameter controlling the amount of non- linearity. The recently proposed AdAct [51] learns weighted combination several activation functions, and DiTAC [9] learns diffeomorphic activation function for CNNs. However, these activation functions are not anput-adaptive. desirable property in GNNs.
Graph Activation Functions. Typically, GNNs are coupled with conventional activation functions [43, 78, 84), which were not originally tailored for graph data, graph tasks, or GNN models. This implies that these activation functions do not inherently adapt the structure of the input graph, which was found to be an important property in other GNN components, such as graph normalization [21]. Recent works have suggested various approaches to bridge this gap. Early works such as Scardapane al. [70] propose learning activation functions based on graph kernels, and lancu et al. [38] introduces Max and Median filters, which operate on local neighborhoods in the graph, thereby offering adaptivity to the input graphs A notable advancement in graph-adaptive activation functions is GReLU [92], parametric piecewise atfine activation function achieving graph adaptivity by lcarning parameters through a hyperfunction that takes into account the node fcatures and the connectivity of the graph. While these approaches demonstrate the potential to enhance GNN performance compared to standard activation functions, they are constrained by their blueprint, often relying on piecewise ReLU composition, which can be performance-liitting [41]. Moreover, a fixed blueprint limits flexibility, i.e., the ability express variety of functions. As we show in Figure 2. attempts to approximate traditional activation functions such as ELU and Tanh using piecewise ReLU composition with different segment counts (K = 1, 2, and 3), reveal limited approximation power. On the contrary, our DIGRAF, which leverages CPAB, yields significantly better approximations. Furthermore, we demonstrate the approximation power of activations learned with the CPAB framework in our DIGRAF in Appendix E.
3

---

0.33
0g-
0.5,
0.5
0g-(-1 -1,
-015-
Tewellation Vertsoes
-333
(a)
(b)
Figure 3: An example of CPA velocity fields o defined on the interval 5 [-5, 5] with a tessellation consisting of five subintervals. The three different parameters, 02. 02. and 0g define three distinct CPA velocity fields (Figure 3a) resulting in separate CPAB diffeomorphisms fo (x) (Figure 3b).
3
Mathematical Background and Notations
this paper, we utilize the definitions from CPAB a framework for efficiently learning flexible diffeo- morphisms [24, 25], alongside basic graph learning notations, to develop activation functions for GNNs. Consequently, this section outlines the essential details needed to understand the foundations of our DIGRAF.
3.1 CPAB Diffeomorphisms
Ω = (a, b] C TR be closed interval, where b. We discretize Ω using tessellation with Wp intervals, which, in practice, is oftentimes an equispaced 1D meshgrid with No segments [25] (see Appendix C for formal definition of tessellation). Our goal in this paper to learn diffcomorphism, activation function. Formally, diffeomorphism is defined follows:
+ Ω that we will
Definition 3.1 (Dificomorphism on a closed interval Q2). diffeomorphism on a closed interval Q C IR is any function f Ω + Q that is (1) bijective, (2) differentiable, and (3) has a differentiable inverse f-1
To instantiate CPAB diffeomorphism parameterized by R-P-11 We display examples of velocity fields n for various instances 0 in Figure 3a demonstrate the distinct influence on o0, Formally, velocity field nº defined as follows:
define continuous piecewise-affin affine (CPA) velocity field e
Definition 3,2 (CPA velocity field o on Q),, Given lessellation P with Ny intervals on closed domain Q, any velocity field 9 Ω R termed continuous and piecewise-afline if (1) o continuous, and (2) v an affine transformation on each interval P.
The CPA velocity field o defines a differentiable trajectory o t) Q x R + S for each & E Q. The trajectories computed by integrating the velocity field o to time i, and used construct the CPAB diffeomorphism. We visualize resulting diffeomorphism in Figure with matching colors denoting corresponding pairs of o and f (z)). Mathematically.
Definition 3,3 (CPAB Diffeomorphism), Given CPA velocity field v., the CPAB diffeomorphism f at point defined as:
(x) (z,t=
such that o0 (x t 1) solves the integral equation:
(o,,))=x) = o ( dr.
(2)
In arbitrary dimensions, computing Definition 3.3 required using ordinary differential equation solver and expensive. However, for diffeomorphisms, as in our DIGRAF, there are closed-form solutions to CPAB diffcomorphism and its gradients [25], offering an efficient framework for learning activation functions.
3.2 Graph Learning Notations
Consider a graph G = (V, F.) with N N nodes, where V = (1...., N} the set nodes and F C V is the set of edges. Let A (0, NXN be the adjacency matrix of G. and X E NxF the node feature matrix,

---

where F is the number of input features. We denote the feature vector of node E V as E RR, which corresponds to the v-th row of X. The imput node features X are transformed into the initial node representations
H E R NxC, using an embedding function emb: : R + IR to X, where C' is the hidden dimension, that is
emb.
(3)
The initial features o are fed GNN comprised of L N layers, where each layer / (I..... L) is followed by activation function o0 (()) R - R, and (()) is set of possibly learnable parameters of ol Specifically, the intermediate output of the 1-th GNN layer is denoted as:
GNN LAYDR (H"-)),)) .A)
(4)
where H ENCC The activation function ( is then applied flement-wise to ( yielding node features A. (00 0 Yu E V. Ve e [C]. Therefore, the application of ( can be equivalently written as:
H
(
(5)
In the following section, we will show how this abstraction is translated to our DIGRAF.
4 DIGRAF
In this section, we formalize our approach, DIGRAF, illustrated in Figure 1, which leverages diffcomorphisms to learn adaptive and flexible graph activation functions.
4.1 A CPAB Blueprint for Graph Activation Functions
Our approach builds on the highly flexible CPAB framework [24, 25] and extends it by incorporating Graph Neural Networks (GNNs) to enable the learning of adaptive graph activation functions. While the original CPAB framework was designed for grid deformation and alignment tasks, typically in ID, 2D, 3D spaces, we propose a novel application of CPAB in the context of learning activation functions, as described below.
DIGRAF, we treat a node feature (single channel) as a one-ooneensee mne-dimenionall (1D) point. Given the node fea- tures matrix H E R NC we apply DIGRAF per en- try in H. in accordance with the typical element- wise computation of activation functions. We mention that, while CPAB was originally designed to learn grid deformations, it can be utilized as an activation func- tion blueprint by considering a conceptual shift that demonstrate Figure 4. Given an input function (shown red in the figure), CPAB deforms grid coor- dinates, i.c., it transforms it along the horizontal axis, as shown in the blue curve. In contrast, DIGRAF transforms the original data points along the vertical axis, resulting in the green curve. This conceptual shift can be seen visually from the arrows showing the different dimensions of transformations. We therefore refer to the vertical transformation of the data as their activations. Formally. we define the transformation function 7'() as the element-wise application of the diffcomorphism fd from Equation (1):
0-
Outpot
(2))(2)) CPAB: (()))
DICRAF: (r. (a(x))
Input
Figure 4: Different transformation strategies. The input function (red), CPAB transformation (blue), and DIGRAF transformation (green), within 52 = [-5, 5| using the same O. While CPAB stretches the input, DIGRAV stretches the output, showcas- ing the distinctive impact of each approach.
(
pol) (AW.C)
(6)
where (( denotes learnable parameters of the transformation function O.) that parameterize the underlying CPA velocity field as discussed in Section 3. In Section 4.2, we discuss the learning of 00) in DIGRAF.
The transformation 7-() Ω Q described in Equation (6) is based on CPAB and therefore takes input values within domain
(a, b). and outputs a value within that domain, where a b are hyperparameters. In practice, we take a = -b, such that the activation function can be symmetric and centered around 0, a property known to be desirable for activation functions [16]. For any entry in the intermediate node features H (Equation (4)) that is outside the domain 22 we use the identity function. Therefore, a DIGRAF activation function reads:
T (Ti)) (()). If Aine Ω
DIGRAF(F. (1)
c.,,, g())
Otherwise
5

---

In practice. DIGRAF is applied element wise in parallel over all entries, and we use the following notation. which yields the ouiput features post the activation of the (-th GNN layer:
H 9 DIGRAF(H ,.
(8)
4.2 Learning Diffcomorphic Velocity Fields
DIGRAF, defined in Equation (7), introduces graph-adaptivity into the transformation function T by employ- ing an additional CNN, denoted GNNaer. that returns the diffeomorphism parameters Ø0)
0 H A) PoOL (GNN (H .A)).
where POOL is a graph-wise pooling operation, such as R-P-11 which is dependent on the tessellation size Np, is then used to compute the output of the I-th layer, H( as described in Equation (8). We note that Equation (9) yields different ( for every input graph and features pair (H)) A), which implies the graph-adaptivity of DIGR AF. Furthermore, since GNN xer is trained with the other network parameters in an end-to-end fashion, DIGR AF is also adaptive to the task of interest. In Appendix B, we provide and discuss the implementation details of GNNacT and Poor..
mean pooling. The resulting vector @
Variants of DIGRAF. Equation (9) describes an approach to introduce graph-adaptivity to / using GNNACT- ACT- An alternative approach is to directly optimize the parameters g()) E R-Vp-1, without using an additional GNN. Note that this case, input and sopp-aaattiiity are sacrificed favor of a computationally lighter solution. We denote this variant of our method by DIGRAF (W/O ADAP.). Considering this variant is important because it allows us to: (i) offer a middle-ground solution in terms of computational effort, and (ii) it allows us to directly quantify the contribution of graph-adaptivity in DIGRAF. In Section 5, we compare the performance of the methads.
Velocity Field Regularization. To ensure the smoothness of the velocity field, which will encourage training stability [81], we incorporate regularization term in the learning procedure of 000, Namely, we follow the Gaussian smoothness prior the CPA velocity field from Freifeld et al. [24]. which was shown be effective in maintaining smooth transformatios. The regularization term defined as follows:
R(100) (1) 1
Σ
EPAA 0
(10)
where Eepa represents the covariance of a zero-mean Gaussian smoothness prior defined as in Freifeld et al. [24]. We further maintain the boundedness of g()) by employing hyperbolic tangent function (Tanh). In this way. (0) remains in 1, 1] when applied in (')) in Equation (7), ensuring that the velocity field parameters remain bounded, encouraging the overall training stability of the model.
4.3 Properties of DIGRAF
In this section, we focus on understanding the theoretical properties of DIGRAF, highlighting the compelling attributes that establish it a performant activation function for GNNs.
DIGRAF yields differentiable activations. By construction, DIGRAF learns diffeomorphism. which is differentiable by definition. Being differentiable everywhere is considered beneficial as allows for smooth weight updates during backpropagation, preventing the zigzagging effect in the optimization process [77].
DIGRAFI is bounded within the input-output domain Q. We point out in Remark D.3 that the diffeomorphism (:;00)) is a 2 transformation. Any diffeomorphism continuous, and by the extreme value theorem, To (1:)0) is bounded in f. This prevents the activation values from becoming excessively large. property linked faster convergence [16].
has the capacity learn activation function governed by a parameters vector ((), DIGRAF can be adjusted through gd) to maintain zero-centered nature. This property associated with accelerated convergence in neural network training [16].
DIGRAF learn to be zero-centered. Benefiting from its flexibility, DIGRAF
activation functions that are inherently zero-centered. As an
DIGRAF is efficient. DIGRAF exhibits linear computational complexity, and can further achieve sub-linear running times via parallelization in practice [25]. Moreover, with the existence
closed-form solution for yo()) and its gradient in the ID case [24], the computations of CPAB can be done efficiently. Additionally, the measured runtimes, detailed in Appendix H, underscore the complexity comparability of DIGRAF with other graph activation functions.

---

In addition to the above properties, which follow from our design choice of learning diffeomorphisms through the CPAB framework, we briefly present the following properties, which are formalized and proven in Appendix D.
DIGRAF is permutation equivariant. We demonstrate in Proposition D.4 that DIGRAF exhibits permutation equivariance to node numbering, ensuring that its behavior remains consistent regardless of the ordering of the graph nodes, which is a key desired property in designing GNN components [8].
DIGRAF is Lipschitz continuous. We show in Proposition D.2 that DIGRAF is Lipschitz continuous and derive its Lipschitz constant. Since it is also bounded, we can combine the two results, which leads us to the following proposition:
Proposition 4.1 (The boundedness of [-; (=) in DIGRAF). Given bounded domain Q = [] C R where a b, and any two arbitrary points x, y E Q the maximal difference of a diffeomorphism (()) with parameter 0 in DIGRAF bounded as follows:
[x ( T': 0(0) S min(|b - al, 1x yl exp(C (oor)) (o(t)))
(11)
where C Cott is the Lipschitz constant of the CPA velocity field o0)))
DIGRAF extends commonly used activation functions. CPAB [24, 251, which used as framework to leam the diffeomorphism DIGRAF, is capable of learning and representing wide range of diffeomorphic functions. When used activation function, the transformation 1 g'0) (=)))))))))))))))))))) in DIGRAF adapts to the specific graph and task by learning different (²) parameters, rather than having fixed diffeomorphism. Examples of popular and commonly used diffeomorphisms utilized as activations include Sigmoid, Tanh, Softplus, and ELU, as show Appendix D. Extending this approach is our DIGRAF that learns the diffeomorphism during training rather than selecting a pre-defined function.
5
Experiments
this section, we conduct an extensive of experiments to demonstrate the effectiveness of DIGRAF as graph activation function. Our experiments soek to address the following questions:
(Q1) Does DIGRAF consistently improve the performance of GNNs compared to existing activation functions on a broad set of downstream tasks?
(Q2) To what extent graph--dapptiity DIGRAF beneficial when compared to our baseline of DIGRAF (W/O ADAP.) and existing activation functions that lack adaptivity?
(Q3) Compared with other graph-adaptive activation functions, how does the added flexibility offered by DIGRAF impact downstream performance?
(Q4) How do the considered activation functions compare in terms of training convergence?
We compare DIGRAF with three categories of relevant and competitive baselines: (1) Standard Activation Functions, namely Identity, Sigmoid [69]. ReLU [26]. LeakyReLU [50]. Tanh [35], GeLU [34], and ELU [12] estimate the benefit of learning activation functions parameters: (2) Learnable Activation Functions, specifically PReLU [33], Maxout [29] and Swish [66]. to assess the value graph-adaptivity; and (3) Graph Activation Functions, such as Max [38], Median [38] and GReL.U [92], to evaluate the effectiveness of DIGR AF's design in capturing graph structure and the blueprint flcxibility of DIGRAF as discussed in Section 4.
Baselines.
All baselines integrated into GCN [43] for node tasks and GIN [84] (GINE [37] where edge features available) for graph tasks, ensure fair and meaningful comparisons, isolating the impact of other design choices. provide additional details on the experimental settings and datasets Appendix G, as well as additional experiments, including ablation studies, in Appendix E.
5.1 Node Classification
Our results are summarized in Table where we consider the BLOGCATALOG [86], FLICKR (86), CITE- SEER [73], CORA [53], and PUBMED [58] datasets. As can be seen from the Table, DIGRAF consistently outperforms all standard activation functions, as well as all the learnable activation functions. Additionally, DIGRAF outperforms other graph-adaptive activation functions. We attribute this positive performance gap to the ability of AF to learn complex non-linearities due to its diffeomorphism- -based blueprint, compared piecewise linear or pre-defined functions as other methods. Finally, we compare the performance DIGRAF and DIGRAF (W/O ADAP.). We remark that in this experiment, we are operating in a transductive setting, as the data consists of single graph, implying that both DIGRAF and DF (W/O ADAP.) are adaptive in that DIGRAF slightly outperforms the DIGRAF (W/O ADAP.) and we attribute this
this case. Still.
7

---

Table 1: Comparison of node classification accuracy (%) f different datasets using various baselines with DIGRAF. The top three methods are marked by First, Second, Third.
Method 4/ Dataset
BLo0 CATALOG
FLICKR
CITESBER
CORA
PURMED
STANDARD ACTIVATIONS
GCN + Ideatity
74.8+055
53.5+1111
69.1=116
80.5+1.2
77.6+2.11
GCN Sigmoid [69]
39.7:44.5
18.3:112
27.9221
32.1+233
5284166
GCN + ReLU [43]
72.1.1.9
50.7.1.33
67.7.2.3
1.
77.6.1.2
GCN + LeakyRel.UU [50]
726622
51.04200
6..42
1.
76.81:
GCN+ [35]
73.910.5
80.1113
GCN + Gel.U/ [34]
75.8::0.55
16
1.7
1.9
77.142277
GCN + ELU [12]
74.8+0.5
53.4111
69.1
80.7+112
77.5+2222
LEARNABLE ACTIVATIONS
$3.2:1.55
69.21:3
80.11122
77.612.11
GCN + Maxout [29]
72.4:1.4
$4.0:118
68.5
7.98
77.3.1.9
GCN + Swish [66]
76.0::077
55.74:
67.7
79.241.11
77.34288
GRAPH ACTIVATIONS
[38]
47.5:1:099
59.71229
76.01
75.0:114
GCN + Median [38]
77.7.10.7
0..
61.3
77.11111
75.7.12.5
GRel.U [92]
73.7
14.4::1.6
68.52
818
GCN + DIGRAF (W/O ADAR.
80.8.1.0.6
68.6.11.8
69.2.12.1
81.11111
78.3.11.6 79.3:114
DIGRAF
81.6:1:0.8
69.6:1:0.6
69.5=114
82.81111
performance gain the GNN layers within DIGRAF that are (i) explicitly grapb-aware, and (ii) facilitate the learning of better diffecomorhism parameters ( (Equation (9)) due to the added complexity.
5.2 Graph Classification and Regression
7IICC-12. In Table we present results on ZINC- 12K [75, 31, 18] dataset for the regression constrained solubility of
Table 2: Comparison ZINC- 12K der the 500K parameter budget. The top
molecules. We note that DIGR AF achieves an MAE of 031302, surpassing the best-performing activation on this dataset, Maxout, by 0.0285, which translates to relative improvement
three methods are First, Second, Third.
18%.
Method
ZINC (MAE L)
OGB bench- mark [36], namely, MOLESOL, MOLTOX21, MOLBACE, and MOL- HIV. The results are reported in Table 3, where it is noted that DIGRAF achieves significant improvements compared to stan- dard, learnable, and graph-adaptive activation functions. For instance, DIGRAF oblains ROC-AUC score 80.28% on MOLHIV, an absolute improvement of 4.7% over the best per- forming activation function (ReLU).
OGB. We evaluate DIGRAF on
STANDARD ACTIVATIONS
Identity
0.2460+00002
GIN Sigmoid
0.3879-10.0058
GIN ReLU [84]
0.1630=10.0040
GIN LeakyRRLU
0,1718::0 0042
GIN Tanh [35]
0.1797:10.0006
Gel.U [34]
0,1896.10.002
GIN ELU [12]
0.0041
LEARNABLE ACTIVATIONS GIN + PReLU [33]
0.179% 0000667
TUDataset... In addition to the aforementioned datasets, we evaluate DIGRAF the popular TUDatasets [56]. We present results on MUTAG, PTC, PROTEINS, NCII and NC1109 in
0.1587-10.0057
GIN Swish [66]
0.1636:10.0039
GRAPH ACTIVATIONS
Table 5 in Appendix E. The results show that DIGRAF always within the top-three performing activations across all datasets. As example, PROTEINS dataset, provement % over the hest-performing activation functions (Maxout and GReLU).
GIN Max [38]
0.1661:20.005
Median [38]
absolute im-
GIN GReL.t/ [92]
0,3003-10.0086
GIN DIGRAF (W/O ADAR.)
0.1382:10.007
GIN DIGRAF
0.1302::0.0090
5.3 Convergence Analysis
Besides improved downstream performance, another important aspect of activation functions is their contribution training convergence [16]. therefore present the training curves of DIGRAF as well as the rest considered baselines to gain insights into their training convergence. Results for representative datasets are presented in Figure 5, where DIGRAF achieves similar or better training convergence than other methods, while also demonstrating better generalization abilities due its better performance.
5.4 Discussion
Our extensive experiments span across 15 different datasets and benchmarks, consisting of both node- and graph-level tasks. Our key takeaways follows:

---

Table 3: A comparison of DIGRAF to natural baselines, standard, and graph activation layers OGB datasets, demonstrating the advantage of our approach. The top three methods are marked by First, Second, Third.
Method L/ Dataset
MOLESOL
MOLTOX21 ROC-AUCT 1
MOLDACE 1
MOLHIV ROC-AUC
STANDARD ACTIVATIONS
GIN Identity
0.002
74.51:20.44
72.69+2.933
75.12=20.77
GIN Sigmoid [69]
0.884=0.043
69.15=00552
68. 7.-43668
73.87=00800
ReLU [84]
74.91-10.51
12.97-44.00
75.58=1.40
GIN LeakyReLL
1.219=0.005
74.60-11.10
73.40:1..19
74.75 1.20
Tanh [35]
1.190=0.044
74.92.12..77
GIN GeLU
1,147=0.050
74.29++0.00 59
75.5 35.39223.32
74.19
GIN ELU (12)
1.104=0.038
75.08++0.62
76.10=t3.29
75.09=0.65
LEARNABLE ACTIVATIONS
[33]
1.06-0
76.16:12.28
GIN Maxxut [29]
1.109=0.045
75.14:4:0.88 87
76.83-4...8 88
72.75 10
GIN + Swish [66]
1.113=0.066
73.31=1100
77.23:+2.35
72.95=0064
GRAPH ACTIVATIONS
Max [38]
1.199
77.04
1.008
GIN Median [38]
1.099000038
74.39:1:0 90
27.26-12.74
72 80-22.22
GReLU [92]
1.108=0.066
75.33=+0.51
75.17:22.60
73.45-+1.
GIN DIGRAF (W/ ADAP.) DIGRAF
0.9011=:0.0047
76,37:2:0:9
18.90-11.41
79.19=11386
0.8196.10.051
77.03:10.59
1.37
Triinieg Loos
Training Loos
Training Loss
19400
Rpach
Epoch
Fpoch
(a) CoRA
(b) FLICKR
(c) ZINC-12K
Figure 5: Convergence analysis DIGRAF compared baseline activation functions. The plot illustrates the training loss over epochs, showcasing the overall faster convergence of DIGRAF.
(A1) Overall Performance of DIGRAF:
The performance offered by DIGRAF is consistent and on par with or better than other activation functions, across all datasets. These results establish DIGRAF as highly effective approach for learning graph activation functions.
(A2) Benefit of Graph-Adaptivity: DIGRAF outperforms the learnable (although not graph-adaptive) activa- tion functions such PReLU. Maxout, and Swish, well as non-graph adaptive baseline DIGRAF (W/O ADAP.), on all considered datasets. This observation highlights the erucial role of graph-adaptivity activation functions for GNNs.
(A3) The Benefit of Blucprint Flexibility: DIGRAP AF consistently outperforms other graph-adaptive activation functions like Max, Median, and GReLU. We tie this positive performance gap to the ability of DIGRAF model complex non linearities due to its diffeomorphism based blueprint, compared to piecewise linear or pre-defined functions as in other methods.
(A4) Convergence of DIGRAF: shown in Section 5.3, addition overall better downstream performance, DIGRAF allows to achieve better training convergence.
In summary, compared with 12 well-known activation functions used in GNNs, and across multiple datasets benchmarks, DIGRAF demonstrates superior, learnable, flexible, and versatile graph-adaptive activation function, highlighting it as strong approach for designing and learning graph activation functions.
Conclusions
In this work, we introduced DIGRAF, novel activation function designed for graph-structure data. Our approach leverages Continuous Piecewise-Arie Affine Based (CPAB) transformations to integrate graph-aud addativee mechanism, allowing DIGRAF to adapt the unique structural features of input graphs. We show that DIGRAF

---

exhibits several desirable properties for activation function, including differentiability, boundedness within defined interval, and computational efficiency. Furthermore, we demonstrated that DIGRAF maintains stability under input perturbations and permutation equivariant, therefore suitable for graph-based based applications. Our extensive experiments on diverse datasets and tasks demonstrate that DIGRAF consistently outperforms traditional, Icarnable, and existing graph-specific activation functions.
Limitations and Broader Impact. While DIGRAF demonstrates consistent superior performance com- pared to existing activation functions, there remain areas for potential improvement. For instance, the current formulation limited to learning activation functions that belong to the class of diffeomorphisms, which, despite encompassing wide range of functions, might not be optimal. By improving the performance on real-world tasks like molecule property prediction, and offering faster training convergence, we envision a positive societal impact by DIGRAF in drug discovery and in achieving lower carbon footprint.
Acknowledgments
BR acknowledges support from the National Science Foundation (NSF) awards, CCF-1918483, CAREER ITS-1943366 and CNS 22S-2212160, Amazon Research Award, AnalytiXIN, and the Wabash Hcartland Innovation Network (WHINN), Ford, NVidia, CISCO, and Amazon. Computing infrastructure was supported in part by CNS-1925001 (CloudBank). This work was supported in part by AMD under the AMD HPC Fund program. ME is funded by the Cavaanik-Cambriee fellowship, the Cambridge Accelerate Programme for Scientific Discovery, and the Maths4DL EPSRC Programme. The authors thank Shahaf Finder, Ron Shapira- -eeber, and Oren Freifeld for the discussions CPAB.
References
[1] Stéphanie Allassonnière, Estelle Kuhn, and Alain Trouvé. Construction of bayesian deformable models via a stochastic approximation algorithm: convergence study. Bernoulli, 2010.
Stéphanie Allassonnière, Stanley Durrleman, and Estelle Kuhn. Bayesian mixed effect atlas estimation with diffcomorphic deformation model. SIAM Journal on Imaging Sciences, 8(3):1367 1395,, 2015.
[3] Andrea Apicella, Francesco Donnarumma, Francesco Isgrò, and Roberto Prevete. survey on modem trainable activation functions. Neural Networks, 138:14-32 2021.
Beatrice Bevilacqua, Moshe Eliasof, Eli Meirom, Bruno Ribeiro, and Haggai Maron. Efficient subgraph gnns by learning effective selection policies. arXiv preprint arXiv:2310. 200820 2023.
[5] Filippo Maria Bianchi, Danicle Grattarola, and Cesare Alippi. Spectral clustering with graph neural networks for graph pooling. In International conference on machine learning, pages 874 883, PMLR, 2020.
[6] Lukas Biewald. Experiment tracking with weights and biases, 2020. URL httpp://ww/ wandb.com/ Software available from wandb.com.
[7] Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? In International Conference on Learning Representations, 2022.
[8] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.3444 2021.
[9] Irit Chelly, Shahaf E Finder, Shira lfergane, and Oren Freifeld. Trainable highly-expresee activation functions. In European Conference on Computer Vision, 2024.
[10] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolu- tional networks. In International conference machine learning, pages 1725-1735. PMLR, 2020.
[11] Amó Clevert, Thomas Unterthiner, and Sepp Hochreiter, Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint
2015.
[12] Amork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus), 2016.
[13] Ingrid Daubechies, Ronald Vore, Simon Foucart, Boris Hanin, and Guergana Petrova. Nonlinear approximation and (deep) relu networks. Constructive Approximation, 55(1):127177,, 2022.
[14] Tim De Ryck, Samuel Lanthaler, and Siddhartha Mishra. On the approximation of functions by tanh neural networks. Neural Networks, 143:732- 2021.
10

---

[15] Nicki Skafte Detlefsen, Oren Freifeld, and Soren Hauberg. Deep diffeomorphic transformer networks. In Proceedingy of the IEEE conference compuler vision and pattern recognition, pages 4403-4412, 2018.
[16] Shiv Ram Dubey, Satish Kumar Singh, and Bidyut Baran Chaudhuri. Activation functions in deep learning: A comprehensive survey and benchmark. Neurocomputing. 503:92- 10092200,,--00 2022.
[17] Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Graph neural networks with learnalec structural and positional representations. In International Conference Learning Representations. 2022.
[18] Vijay Prakash Dwivedi, Chaitanya K Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. Journal of Machine Learning Research, 24(43):1-45 2023.
[19] Stefan Elfwing, Eiji Uchibe, Kenji Doya. Sigmoid weighted linear units for neural network function approximation reinforcement learning. Neural networks, 107:3-11, 2018.
[20] Moshe Eliasof, Fabrizio Frasca, Beatrice Bevilacqua, Eran Treister, Gal Chechik, and Haggai Maron. Graph positional encoding via random feature propagation. In International Conference on Machine Learning, pages 9202-9223, PMLR, 2023.
[21] Moshe Eliasof, Beatrice Bevilacqua, Carola- Bibblae Schönlich, and Haggai Maron. Granola: Adaptive normalization for graph neural networks. arXiv preprint arXiv:2404.13344 2024.
[22] Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric, 2019.
[23] Fabrizio Frasca, Beatrice Bevilacqua, Michael M Bronstein, and Haggai Maron. Understanding and extending subgraph gnns by rethinking their symmetries. Advances in Neural Information Processing Systems, 2022.
[24] Oren Freifeld, Soren Hauberg, Kayhan Batmanghelich, and John Fisher. Highly-expresiive spaces of well-behaved transformations: Keeping it simple. In Proceedings of the IEEE International Conference on Computer Vision, pages 2911 1-2919, 2015.
[25] Oren Freifeld, Soren Hauberg. Kayhan Batmanghelich, and Jonn W Fisher. Transformations based on continuous piecewise-ane affine velocity fields. IEEE transactions on pattern analysis and machine intelligence. 39(12)24996 2509, 2017.
[26] Kunihiko Fukushima. Visual feature extraction by multilayered network of analog threshold elements. IEEE Transactions on Systems Science and Cybernetics, 5(4):322-33) 333, 1969.
[27] Hongyang Gao and Shuiwang Ji. Graph u-nets. In international conference machine learning, pages 2083-2092. PMLR, 2019.
[28] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learming, pages 1263 1272. PMI.R, 2017.
[29] lan Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 Proceedings of Machine Learning Research, pages 1319-1327, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
[30] Thomas Hakon Gronwall. Note on derivatives with respect to a parameter of the solutions a system of differential equations. Annals of Mathematics, pages 292-296, 1919.
[31] Rafael Bonnnnzll, Jennifer N. Wei, David Duvenaud, José Miguel Hemández-Lohate, Benjamín Sänchez-Lengeling, Dennis Sheberla, Jorge
Timothy D. Hirzel, Ryan P. Adams, and Alán Aspuru- Guzik. Automatic chemical design using data-driven continuous representation of molecules. ACS Central Science, 4(2):268 226-6667766 January 2018. ISSN 2374-7951. doi: 10. 10.1021/acccentsii.70000 7b005722.
[32] Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse problems, 34(1):
014004, 2017.
[33] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages
2015.
[34] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), 2023.

---

[35] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural compulation. 1735:1735-1780. 1997.
[36] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint (eXiv:2005.006880 2020.
[37] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitmik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategics for pre-training graph neural networks. In International Conference on Learning Representations, 2020.
[38] Bianca Iancu, Luana Ruiz, Alejandro Ribeiro, and Elvin Isufi. Graph- adaptive activation functions for graph neural networks. 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MISP), pages 1 1-6, IEEE, 2020.
[39] Prateck Jain, Purushottam Kar, et al. Non-convex optimization for machine learning. Foundations and Trends in Machine Learning, 106--4):12-363, 2017.
[40] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figumov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Amna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873) 2021.
[41] Sammy Khalife and Amitahh Basu. On the power of graph neural networks and the role of the activation function, 2023.
[42] Dongkwan Kim and Alice Oh. How find your friendly neighborhood: Graph attention design with self-supervision. International Conference on Learning Representations, 2021.
[43] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations. 2017.
[44] Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. Advances Neural Information Processing Systems, 34: 21618-21629, 2021.
[45] Vladimír Kunc and Jiff Kléma. Three decades of activations: A comprehensive survey of 400 activation functions for neural networks. arXiv preprint arXiv:2402.09099, 2024.
[46] Junhyun Lee, Inyeop Lec, and Jaewoo Kang. Self-attention graph pooling. In International conference on machine learning, pages 3734-3743, PMLR. 2019.
[47] Qimai Li. Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-superviied learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
[48] Shan Sung Liew, Mohamed Khalil-Hani, and Rabia Bakhteri. Bounded activation functions for enhanced training stability of deep neural networks on visual pattern recognition problems. Neurocomputing, 216: 718-734, 2016. ISSN 0925-2312. doi: https://doi.or/10..000 wtl.jjjcccm..216.60
[49] Ziming Liu, Yixuan Wang. Sachin Vaidya, Fabian Ruchle, James Halverson, Marin Soljačić, Thomas Y. Hou, and Max Tegmark. Kan: Kolmogorov- networks, 2024.
[50] Andrew Maas. Rectifier nonlinearities improve neural network acoustic models. International conference on machine learning, 2013.
[51] Ritabrata Maiti. Adact: Learning to optimize activation function choice through adaptive activation modules. In The Second Tiny Pupers Track at ICLR 2024, 2024.
[52] nigo Martinez, Elisabeth Viles, and Igor G Olaizola. Closed-form diffcomorphic transformations for time series alignment. In International Conference on Machine Learning, pages 15122-1.5158. PMI.R, 2022.
[53] Andrew Kachites McCallum, Kamal Nigam, Jason Rennic, and Kristic Scymore. Automating the construc- tion of internet portals with machine learning. Information Retrieval, 3:127-163, 2000.
[54] Akash Mishra, Pravin Chandra, and Udayan Ghose. A non-monotonic activation function for neural networks validated on benchmark tasks. In Modern Approaches in Machine Learning and Cognitive Science: A Walkthrough: Latest Trends in Al. Volume 2. pages 319-327. Springer, 2021.
12

---

[55] Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 4602-4609, 2019.
[56]
Christopher Morris, Nils M. Kriege. Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs, 2020.
[57] Christopher Morris, Yaron Lipman, Haggai Maron, Bastian Ricck, Nils M. Kriege, Martin Grohe, Matthias Fey, and Karsten Borgwardt. Weisfeiler and leman go machine learning: The story so far. Journal of Machine Learning Research, 24(333):1 59, 2023.
[58] Galileo Namata, Ben London, Lise Getoor, Bert Huang, and U Edu. Query-driven active surveying for collective classification. In 10th international workshop on mining and learning with graphs, volume 8, page 1, 2012.
[59] Emmanuel Noutahi, Dominique Beaini, Julien Horwood, Séhastien Giguère, and Prudencio Tosson. Towards interpretable sparse graph representation learning with laplacian pooling. arXiv preprint arXiv: 1905. 1577, 2019.
[60] Chigozie Nwankpa, Winifred Ijomah, Anthony Gachagan, and Stephen Marshall. Activation functions: Comparison of trends in practice and research for deep learning. arXiv preprint arXiv: 0tf:::/8/11.33777.. 2018
[61] Joost AA Opschoor, Philipp C Petersen, and Christoph Schwab. Deep relu networks and high-order finite element methods. Analysis and Applications, ((05):715-770,, 2020.
[62] Abhishek Panigrahi, Abhishek Shetty, and Navin Goyal. Effect of activation functions on the training of overparametrized neural nets. arXiv preprint 1908 2019.
[63] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- perffrrerormeee deep learning library, 2019.
[64] Ilan Price, Nicholas Daultry Ball, Samuel CH Lam, Adam C Jones, and Jared Tanner. Deep neural network initialization with sparsity inducing activations. arXiv e-prints, pages arXiv- 2024.
[65] Omri Puny, Derek Lim, Bobak Kiani, Haggai Maron, and Yaron Lipman. Equivariant polynomials for graph neural networks. In International Conference on Machine Learning, pages 28191-28222. PMI.R, 2023.
[66] Prajit Ramachandran, Barret Zoph, and Quoc Le. Searching for activation functions. arviv preprint arXiv:1710.05944 2017.
[67] Ladislav Rampášek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems, 35:14501-14515. 2022.
[68] Patrick Reiser, Marlen Neubert, André Fberhard, Luca Torresi, Chen Zhou, Chen Shao, Houssam Metni, Clint van Hoesel, Hemrik Schopmans, Timo Sommer, et al. Graph neural networks for materials science and chemistry. Communications Materials, 3(1):93, 2022.
[69] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back- propagating errors. nature, 6238)883)533--66 1986.
[70] Simone Scardapane, Steven Van Vaerenbergh, Danilo Comminiello, and Aurelio Uncini. Improving graph convolutional networks with non-meraetriic activation functions. In 2018 26th European Signal Processing Conference (EUSIPCO), pages 872 176. IEEE, 2018
[71] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner. and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61-88) 80, 2008.
[72] Christoph Schwab, Andreas Stein, and Jakob Zech. Deep operator network approximation rates for lipschitz operators. arXiv preprint arXiv:2307.00 09835, 2023.
[73] Prithviraj Sen. Galileo Namata, Mustafa Bilgic. Lise Getoor, Brian Galligher, and Tina Eliassi-Ras Rad. Collective classification in network data, Al magazine, 29(3):93 93, 2008
[74] Ron A Shapira Weber, Matan Eyal, Nicki Skafte, Oren Shriki, and Oren Freifeld. Diffeomorphic temporal alignment nets. Advances in Neural Information Processing Systems, 32, 2019.
13

---

[75] Teague Sterling and John J. Irwin. ZINC 15 ligand discovery for everyone. Journal of Chemicul Information and Modeling, 55(11))2324-2337, 11 2015. doi:
[76] Ganesh Sundaramoorthi and Anthony Yezzi. Variational pdes for acceleration on manifolds and application to diffeomoophissms. In S. Bengio. H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[77] Tomasz Szandala. Review and comparison of commonly used activation functions for deep neural networks. Bio Bio-inspinn inspired neurocomputing, pages 203 224, 2021.
[78] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint 1rXiv:1110.10000 2017.
[79] Hexiang Wang, Fengqi Liu, Qianyu Zhou, Ran Yi, Xin Tan, and Lizhuang Ma. Continuous piecewise-affine based motion model for image animation. arXiv preprint arXiv:2401.09146 2024.
[80] Yu Guang Wang. Ming Li, Zheng Ma, Guido Montufar, Xiaosheng Zhuang, and Yanan Fan. Haar graph pooling, International conference on machine learning, pages 9952-9962. PMLR, 2020.
[81] Ron Shapira Weber and Oren Freifeld. Regularization-fiee diffcomorphic temporal alignment nets. In International Conference on Machine Learning, pages 30794-308222 PMLR, 2023.
[82] Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. Graph neural networks in recommender systems: survey. ACM Computing Surveys, 55(5):1-37, 2022.
[83] Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolu- tional network, 2015.
[84] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.
[85] Zhen Xu, Xiaojin Zhang, and Qiang Yang. Tafs: Task-aware activation function search for graph neural networks. 2023.
[86] Renchi Yang, Jieming Shi, Xiaokui Xiao, Yin Yang, Sourav Bhowmick, and Juncheng Liu. Pane: scalable and effective attributed network emhedding. The VLDB Journal, 3266)1237 March 2023. ISSN 0949-877X. doi: 10. 00077/00777-022000077
[87] Zhitao Ying. Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec, Hierarchical graph representation learning with differentiable pooling. Advances in neural information processing systems, 31, 2018.
[88] Bohang Zhang, Guhao Feng, Yiheng Du, Di Hc, and Liwei Wang. A complete expressiveness hicrarchy for subgraph gnns via subgraph weisfeiler-lehman tests. In International Conference on Machine Learning, 2023.
[89] Bohang Zhang, Shengjie Luo, Liwei Wang, and Di He. Rethinking the expressive power of gnns via graph biconnectivity. arXiv preprint arXiv:2301.09505, 2023.
[90] Miaomiao Zhang and P Thomas Fletcher. Bayesian statistical shape analysis on the manifold of diffeo- morphisms. Algorithmic Advances in Riemannian Geometry and Applications: For Machine Learning. Computer Vision, Statistics, and Optimization, pages 1-23, 2016.
[91] Xiao-Meng Zhang, Li Liang, Lin Liu, and Ming-Jing Tang. Graph neural networks and their current applications bioinformatics. Frontiers in genetics. 12:690049, 2021.
[92] Yifei Zhang, Hao Zhu, Záqiao Meng, Piotr Koniusz, and Irwin King. Graph-adaptive rectified linear unit for graph neural networks. In Proceedings of the ACM Web Conference 2022, pages 1331 1331,1339 2022.
[93] Zhen Zhang, Jiajun Bu, Martin Ester, Jianfeng Zhang, Chengwei Yao, Zhi Yu, and Can Wang. Hierarchical graph pooling with structure learning. arXiv preprint 1, 2019.
14

---

A Additional Related Work
Graph Neural Networks. Graph Neural Networks [71] (GNNs) have emerged as transformative approach in machine learning, notably following the popularity of the pessagg-passing scheme [28]. GNNs enable effective learning from graph-structured data, and can be applied to different tasks, ranging from social network analysis [43] to bioinformatics [40]. In recent years, various GNN architectures were proposed, aiming to address various aspects, from alleviating oversmoothing [10], concerning attention mechanisms in the message passing scheme [78. 7. 42]. focusing on the expressive power of the architectures [57, 23, 88, 89, 65, 4], given that passsage-passing based architectures are known to bounded by the WL graph isomorphism test [84, 55).
Despite advancements, the poor performance of deep GNNs has led a preference for shallow architectures GCNs [47]. To enhance performance, techniques such as pooling functions have been proposed, introducing generalization by reducing feature map sizes [93]. Methods such as HGP-SL [93], GraphUNet [27]. and LaPool [59] introduce pooling layers specifically designed for GNNs. Beyond node feature, the importance of graph structure and positional features is increasingly recognized, with advancements such as GraphGPS [67] and SAN [44] integrating positional and structural encodings through attention-based mechanisms.
Evaluation of Rectified Activation Functions. Rectified activation functions, represented by the Recti- fied Linear Unit (ReLU), have been widely applied and studied in various neural network architectures due to their simplicity and effectiveness [60, 3, 45J. The prevalent assumption that ReLU's performance is predomi- nantly due to its sparsity is critically examined by Xu et al. [83], suggesting introducing non-zero slope in the negative part can significantly enhance network performance. Extending this, Price et al. [64] investigates sparsity-inducing activation functions, such as the shifted ReL.U, in network initialization and early stages of training. These functions can mitigate overfitting and boost model generalization capabilities. Conversely, it was shown that in overparameterized networks, smoother activation functions, like Tanh and Swish, can enhance the convergence rate, in contrast to non- non-smooth characteristies of ReLU [62]. However, the fixed nature of ReLU and many of its variants restricts their ability to adapt the input, resulting in limited power to capture dynamics in learning.
Advancements in Learnable Activation Functions. Recent research has increasingly focused on adaptive and learnable activation functions, which are optimized alongside the learning process of the network. The AdAct framework [51] introduces learnability by combining multiple activation functioon into a single module with learnable weighting coefficients. However. these coefficients are fixed after training. limiting the framework's 's adaptability to varying imputs. A concurrent work by Liu et al. [49] introduces Kolmogorov- Amold Networks (KAN), novel architecture that diverges from traditional Multi-Layer Perceptron (MLP) configurations, which applics activation functions to network edges instcad of nodes. Unlike our current work, which focuses only on the design of activation functions for GNNs, their research extends beyond this scope and considers a fundamental architecture design. Finally, the recently proposed TAFS [85] learns a task-adaptive (but not graph-adaptive) activation function for GNNs through a bi-level optimization.
B Implementation Details of DIGRAF
Multiple Graphs in one Batch. Consider set of graphs S = (G1, Ga.i.,,..... ... with batch size of B. Let Ns = Ni + No + ..... + NB represent the cumulative number of nodes across the graph dataset. The term Nuxx A max(NN, ,2,...,N denotes the largest node count present in any single graph within S.
To create a unified feature matrix for that encompasses all graphs in the batch, we standardize the dimension by padding each feature matrix X, E N E [B] for graph G. E S from N, to Nuax with zeros. The combined feature matrix X5 is constructed by concatenating the transposed feature matrices X Vi E [B]. resulting in matrix that lies in the domain R (RF-C) Nua. This matrix is permutation invariant; while relabeling nodes changes the row indices, it does not affect the overall transformation process. Therefore. DIGRAF can handle multiple graphs in a batch. In practice, to avoid the overhead of padding, we use the batching support from Pytorch- Gerreerre [22].
Implementation Details of GNNACT In Section 4.2, examined two distinct approaches to learn the diffeoomrphism parameters ((), either directly or through GNN.cr. As shown in Appendix C, llh) determines the velocity field o(0 Predicting graph-dependent (( adds graph-adaptivity to the activation function T'))) In DIGRAF achieve this by employing another GNN GNN described below.
The backbone of GNN,ccr utilizes the same structure as the primary network layers GNN that is, GCN [43] GIN [84]. It is important to note, that while GNN has similar structure to the primary network GNN with ReLU activation function, has its set of learnable weights, and it is shared among the layers. LAYER The hidden dimensions and the number of layers of GNN ACT unlike the primary GNN layers (
15

---

hyperparameters. The weight parameters of GNN are trained concurrently with the main network weights. As described in Equation (9), after the computation of GNN pooling layer denoted by PoOL is placed to aggregate node features. This aggregation squashes the node dimension such that the oulput is not dependent on the specific order of nodes, and it yields the vector of parameters (()
Rescaling H Following the implementation of Freifeld et al. [24], the default ID domain for CPAB set (0, 1J. To enhance the flexibility of T and ensure its adaptability across various input datascts, DIGRAF extends the domain to aa, bj C with a < b as shown Section 3.1. To match the two domains, we rescale the intermediate feature matrix H from S to the unit interval [0, 1] before passing to TW, Let
r = A- 2 then rescaling is performed using the function f f(x) = (ær + r) (2r). Data points outside this range will retain their original value, effectively acting as identity function outside the domain f.
Training Loss Function. As described in Equation (10), we employ a regularization term for the velocity field to maintain the smoothness of the activation function. To control the strength of regularization, we introduce hyperparameter A. We denote CrASK the loss function of the downstream task (i.e. cross-entropy loss in case of classification and mean absolute error in case of regression tasks), and the overall training loss of DIGRAF, denoted as Crow. is given as
LTOTAL
-rASK + ((())))))))
(12)
C Overview of CPA Velocity Fields and CPAB Transformations
In this Section, we drop the layer notations / for simplicity. In Section 3.1, we introduce the concept of a diffeomorphism on a closed interval in Definition 3.1, which can be learned through the integration of a Continuous Piecewise Affine (CPA) velocity field. As detailed in Definition 3.2, the velocity field o is governed by the parameter A and the tessellation 'P. We now discuss how the velocity fields are computed following the methodologies presented by Freifeld et al. [24, 25] and highlight the relations between v. 0 and P. We start by formally defining the tessellation on Q::
Definition C.1 (Tessellation of a closed interval [24). A tessellation P of size Np subintervals of a closed interval Ω [a. b] in IR is partitioning #1],) - that satisfies the following properties:
(1) To a and 2N²
(2) Each point x E Ω lies in at least one subinterval [ut, (444]
(3) The intersection of any two subintervals (m4, 2s41) and [1141, is exactly (1+11}
Ny 1 [2,, x1+1] = ?
(4)
The vector of parameters 0 is linked to the subintervals in P. whose dimension is determined by the number of intervals Nyp Similar to Freifeld et al. [24], we impose boundary constraints that mandate the velocity at the boundary of the tessellation to be zcro, i.c., (0) (1) = 0. This boundary condition allows compose the diffeomorphism in the domain 92 with an identity function for any values outside the domain. Under this constraint, the degrees of freedom (number parameters) for is No
The velocity field then defined as follows:
Definition C.2 (Relation between @ and v., taken from Freifeld et al. [25]), Given tessellation P with Wip intervals on closed domain Ω (a, b). as defined in Definition C.1. Given parameter 0 E R-Vp-1 and an arbitrary point i within the domain, continuous piecewise-afin affine velocity field o can present as follows:
v'(x) = Ny
,,b,I.
(13)
where (b,),Kn Wy- is an orthonormal basis of the space of velocity fields V. such that o E V. and ã
The orthonormal basis ( Wr-2 for the velocity field can be obtained through Singular Value Decomposition
of L. Note that L is matrix constraining the coefficients of each continuous sioeewiss-affin velocity function by ensuring that the velocity value the shared endpoints is the same [52]. Let vec(A) be column vector containing the coefficients for each interval, for instance, [ao. bo, a1 b, for consecutive intervals interval 0 and interval 1. The shared endpoint is 21. To achieve the constrain, we have the equation
ag + bo + a1 (-21) + b1 (-1) O. In this example, the constrain matrix L is L [æ1, 1, -21, -1].
16

---

Table 4: A summary of the properties of activation functions. studied in the corresponding Median-of-medians algorithm can achieve linear time complexity on average.
paper.
Act
Boundedness
Differentihiilit
Linear Complexity
Permutation Equiv. Lipschitz Cont.
Graph Adap.
ReL.U [84]
Tanh [35]
PReLU 1331
Swish [66]
Max [38]
X
Median [38]
GRel. [92]
DIGRAF
By generalizing the previous example, the constraint can be expressed as L + vec(A) = 0 As the endpoints are decided by the tessellation setup, can build the constrain matrix L without knowing pee(A), And thus, orthonormal basis (b, 1y=0 N2 can be computed by giving tessellation setup.
Proposition C.3 (DIGRAF has closed form solution). Equation can be expressed an equivalent ODE.
By allowing 2 to vary and fixing t, the solution to this ODE can be written as a composition of a finite number of solutions w:
8 (xx t) = (1om o mm-1 1.... 2o.c o 15 (x))
o..-
Here m represents the number of cells visited. Given 0, time t, and the smallest cell index containing 2, compute each Vo (x), E E closed-form solution.
{I...., m. from wo (x) Von (x). In other words, DIGRAF has
Proof. The proof follows the steps in Martinez al. [52]. Equation (2) can he expressed the equivalent ODE: (s,t) (t (x, L)). By allowing to vary and fixing l, the solution to this ODE can be written as composition of a finite number of solutions x/:
(x,,
8 (xr, t) = (W6t.... @..em-
2e o d. (x)
Given x, 0, time I, and a function 7 that returns the smallest cell index containing æ. namely c = 7xx). then we compute each vif (z), E (1, ( m} and them in the closed form solution for the ODE. The cell boundary is determined based on the velocity value v(x) point If e(x) 2 0, Ie is the largest point in the interval; otherwise, it is the smallest point. In this setup, cell is ID interval with endpoints. At the hitting time thee, the (2t, tace) is
3 (xx, haa) Tes
where Lh.5 oi log with index c, has coefficients a (slope) and be (bias). These be computed given 0. If th.. then (x, = (2, t). Otherwise, we repeat process with updated values based on the sign v(x).
aga+to The CPAB velocity field continuous piecewise-affine, and each interval
that x Fe. and adjusted
This iterative process continues until convergence, with an upper bound for m being mar(C,, Np G2 + 1). where refers to the first visited cell index, and Nr is the number of closed intervals in the space Q. With the above steps, we can precisely compute each uig (x), i E (L...., m} from vot., (x) viti (z) following the
equation ø" (zz, f) = (15 .... whi,, (x). This allows us to determine exact solution for & (a, t).
D Properties and Proofs
We present a summary of the properties offered by our DIGRAF that are absent in general-purpose purpose activation functions or existing graph activations in Table 4.
Similar to Appendix C, for simplicity, in this Section, we drop the layer notations 1.
In this section, we present the propositions and proofs for the properties outlined Section 4.3. We begin by remarking that shown in Section 4.3, DIGRAFES is bounded within the domain S [a, b). where < b by construction. We then present Proposition D. that outlines the Lipschitz constant of the velocity field o followed by Proposition D.2, showing that DIGRAF is also Lipschitz continuous, and provide an upper bound on its Lipschitz constant.
17

---

Proposition D.1 (The Lipschitz Constant of )²). Given two arbitrary points x. y. E R. and velocity field
parameters RR-r- 11 that define the continuous piecewise-affine velocity field v there exists Lipschitz constant C. = 2z=
NNp- 10,1 such that
(x) v' (v) - C, (l@
8) 2,
(14)
where| |-
and || ||a denote the absolute value of a scalar and the 82 norm of vector, respectively.
First, we note that it was shown in Freifeld et al. [24, 25] that 8 Lipschitz continuous, and now we provide a derivation of that Lipschitz constant. Following Definition C.2, the velocity field v is defined as
Proof.
(w) Σmz=2 oy where (b, [b, p is an orthonormal basis of velocity space. By the definition of
(z) and (y). we have the following:
Np-2
No 2p-2
(x)) -r0(0)
=
Σ
0,
Σ
,,
(15)
Np 0
b, ( 5)
(16)
10 0,]1|b/|l2||[@ ~)1l
(17)
Np-2
Σ
(18)
Np-2
= ( 1)|l2
Σ
10,1
(19)
j=0
C.. (1 1)1
(20)
where the transition between Equation (16) and Equation (17) follows from the triangle inequality, and the transition between Equation (17) and Equation (18) follows from b, being an orthonormal vector.
From the derivation above, and the fact that we know from Freifeld et al. [24, 25] that the velocity field is
Lipschitz continuous, we conclude that the Lipschitz constant C. of 60 reads C.. Σ 10,1-
Xp
Given the Lipschitz constant C.o for e², we proceed to demonstrate that the transformation I (=;0) in DIGRAF is Lipschitz continuous, as well as bounding its Lipschitz constant.
Proposition D.2 (The Lipschitz Constant of DIGRAF). The diffeomorphic function T(:; in DIGRAF defined in Equation (6) for a given set of weights O. which in turn define the velocity field v. Let x. y E IR be two arbitrary points, then the following inequality is satisfied:
(xx; 0) - T(y: 0)1 S 1 -l exp(C,..
(21)
where C.e is the Lipschitz constant of vo
Proof. We begin by substituting T(: 0) with Equation (1) and Equation (6). Utilizing Proposition D.1, we then
establish an upper bound for [T (a; 0) T(y; 0) as follows:
[x:; 0) T(y: 0)1 = |r + j (c dr
(0)(())))) drl
- x
(22)
$ | - vl Co C. / /100/007- 6 (6,,)))
(23)
- exp(
(24)
where C.e is the Lipschitz constant of 8 (Proposition D.1) and the last transition follows from Grönwall's inequality [30]. Consequently, the Lipschitz constant of DIGRAF is bounded from above by (Cp(C...). ). n
Now that we established that [-; 0) is Lipschitz continuous and presented an upper bound, we investigate what is the maximal difference in the output of (-; 0) with respect to two arbitrary inputs 2, y E Ω. and whether it can be bounded. To address this, present the following remark:
18

---

Remark D.3. Given bounded domain f [a. b], < b. by construction, the diffeomorphism T:: 0) with parameter 0 in DIGRAF, in Equation (7), is 12 f2 transformation [24, 25J. Therefore, by the max value theorem, the maximal output discrepancy for arbitrary a,y E 2 is |6 a/, ie. [T(x;; 0) T(y: 0) - 16 - al.
Combining the Proposition D.1, Proposition D.2 and Remark D.3, we formalize and prove the following proposition:
Proposition 4,1 (The boundedness of (:: (1):0)) in DIGRAF). Given bounded domain Q = [a,, b] C R where a < b, and any two arbitrary points J. y E 22 the maximal difference of a diffeomorphism T(: with
parameter 9 DIGRAF is bounded as follows:
(x::0%) Ty: 0 S min((b a), |r ya exp(C (0lt))
(1I)
where Coois is the Lipschitz constant of the CPA velocity field e o00)
Proof. Proposition D.2 we presented an upper bound on the Lipschitz constant of T(::)); and in D.3 we also presented an upper hound on the maximal difference hetween the application of T(:; 0) on two inputs x, y- Combining the two bounds, we get the following inequality:
[ (xx; 0) 7 [y: 0)1 I min((e (6- - al, le yl exp(C.,)).
(25)
The result in Proposition 4.1 gives tighter upper bound the boundedness of the transformation T(: 0) in our DIGRAF that related both to the hyperparameters a, as well as the learned velocity field parameters
Next, we discuss another property outlined in Section 4.3, demonstrating that DIGRAFis is permutation equivariant - a desirable property when designing GNN component [8].
Proposition D ( DIGRAF is permutation equivariant.). Consider a graph encoded by the adjacency matrix E RRNNN where the number of nodes. Let ( E NxC be the intermediate node features at layer I, before the element-wise application of our DIGRAF. Let P be an N x N permutation matrix. Then,
DIGRAF(PH
0.9) P DIGRAF(H .())
(26)
where @ and 9t) are obtained by feeding (H and (() rexpectively, Equation (9).
Proof. We break down the proof into two parts. First, we show that GNNAcT outputs the same 0 under permutations, that is we show
0 (t)
Second, we prove that the activation function ( is permutation equivariant, ensuring the overall method maintains this property.
To begin with, recall that Equation (9) composed by GNN which is permutation equivariant, and by a pooling layer, which is permutation invariant. Therefore their composition is permutation invariant, that is 0 (())
Prior the activation function layer T H undergoes rescaling described in Appendix B, which is permutation equivariant as it operates alement-wwiee. Finally, since activation function ( acts element-wise, and given that 0 remains unchanged, the related CPA velocity fields are identical, resulting in the same transformed output for each entry, despite the entries being permuted (H(). Therefore, DIGRAF is permutation equivariant.
D.1 Diffeomorphic Activation Functions
this section, we provide several examples of popular and well-known diffeomorphic functions, contributing to our motivation to utilize diffeomorphisms as a blueprint for learning graph activation functions. We remark that, differently from standard activation functions, our DIGRAF does not need to follow predefined, fixed template, but can instead learn a diffcomorphism hest suited for the task and input, as ( within CPAB can represent wide range of diffeommrphisms [24, 25].
We recall that, as outlined in Section 3.1, a function is classified as a diffcomorphism if it is (1) bijective, (2) differentiable, and (3) has differentiable inverse.
Sigmoid. We denote the Sigmoid activation function
R - (0, 1), defined by
1
(4)
I I+e e-=
19

---

To prove that a is diffeomorphism, we first establish its bijectivity. Injectivity follows from observing that for any distinct points i1 and i2 in I2, ((m) I+e- can only equal a (22) I+t
if and only if an u2-
For surjectivity, we represent for every E (0, 1) there is an element æ E R such that (z) = To demonstrate differentiability, we examine the derivative a. The derivative
a function of y, such that y 1+e
1-11
ensuring that
d(x) (1 o())),
dæ
which is continuous. Additionally, the inverse function
( In
also bijective and diffcrentiable. Thus, with all these requirements satisfied, a is indeed a diffeomorphism.. Tanh. The hyperbolic tangent function
Lanh(x) e + em
diffcomorphism from R (-1, 1). To establish this, we demonstrate that tanh is bijective and diffcrentiable, with a differentiable inverse function.
Firstly, tanh is injective because tanh(x1) = tanh( then #1 = 22. is also surjective because for any y E (-1, 1), there exists In L+y 1-w
such that tanh(r)
The derivative
d tanh(x) 1 tanh (x)
de
continuous and positive. Additionally, the inverse function
tanh (v) = In [
continuously differentiable. Therefore, tanh qualifies diffeomorphism. Softplus. To establish the Softplus function
softplu(z:) = In(1 + e*)
as a diffeomorphism from to (0, oc), we first demonstrate its injectivity and surjectivity.
Assuming soft = (22) we obtain = e²² implying #1 = 22, hence establishing injectivity. For any y E (0, oo), we find an x E IR such that = In(1 + E"), ensuring surjectivity.
The derivative of the Softplus function,
d softplus(z) 1+ oxx),
dr
where (ar) is the Sigmoid function, known be continuous and differentiable. Therefore, softplus(z:) is continuously differentiable.
Considering the inverse of the Softplus function,
softplus Soauusssss-s (Vy In(e" 1);
its derivative is
softplus (wm e
which continuous for all y >> O, indicating that softplus -1 (y)) is continuously differentiable for all y Therefore, we conclude that softplus function qualifies as diffeomorphism.
ELU. The ELU activation function [11] is defined as below:
if x > 0
BLU(<) a(c" -1
if
x 5
where E IR is constant that scales the negative part of the function. To demonstrate that ELU is bijective, analyze its injectivity and surjectivity. For ELU acts as identity function, which inherently injective.
For æ 5 0, a(e") -1) = a(c²2 1), implies æ1 22. The inverse function for ELU is given by:
EL.U
y In(2 +1)
if
if
20

---

This inverse maps every value in the codomain back to unique value in the domain, proving that ELU is surjective.
Next, examine the continuity of ELU. At 0. ELU( limits for both sides of For 0, limm ELU(x) lim x 0, while for O, we have limg- (LU(z) = -im x<(e - 1) Since both limits are equal, the ELU function is continuous at
a(e" 1) O. Next. check the
0. For the derivative of ELU, i.c.
i 0
ELU(x)
dr
ae 5
at x = O, we have A BLU(s) = ac a. By setting a = 1, the derivative at r = matches the derivative for de >> 0, making the derivative continuous.
The derivative for the inverse function is
y 0
ELU (v)
dy
which is also continuously differentiable. Hence, ELU is diffcomorphism.
E
Additional Results
E.1 Function Approximation with CPAB
ENGRAF
BALE TwsH
10
3313
Figure 6: The approximation error of the Peaks function (Equation (27)) with ReLU, Tanh, and DIGRAF.
The combination learned linear layers together with non- non-lincar functions such as ReLU and Tanh are well- known to yield good function approximations [13, 14). Therefore, when designing an activation function blueprint, i.c., the template by which the activation function learned, it is important consider its approxi- mation power. In Section and in particular Figure 2, demonstrate the ability of the CPAB framework to approximate known activation functions. of CPAB as framework for learning activation functions, leading to our DIGRAP To this end, we consider the ability of multilayer perceptron (MLP) with various activation functions (ReLU, Tanh, and DIGRAF) to approximate the well- 'peaks' function that mathematically reads:
now show additional evidence for the flexibility and power
g y) 3(1-2)² (aa-===e<xpp(-ee (app((x")-(y++++
-10r - - exp( exp(-2²= x² 5²) exp(- exp(-(x+1)"-g") (27)
The peaks function Equation (27) is often times used to measure the ability of methods to approximate functions [32], where the input is point pairs (x, y) E R²2 and the goal is to minimize the mean meea-squared -qurred-error error between the predicted function value g and the actual function value Formally, consider the following MLP:
((x,y) = (α<a<((,)) v) W.),s,
(28)
where a the activation of choice (ReLU, Tanh, or DIGRAF), and Wi E 2x66 W2 E 6444664 W3 E R 64x1
are the trainable parameter matrices of the linear layers in the MLP. The goal, as discussed above, is to minimize the loss [], y) g(z, y) ||2. for data triplets (2;, Vi. g(x,, sampled from the peaks function. our experiment, we sample 50,000 points, and report the obtained approximation error in terms MSE Figure 6. As can be seen, our DIGRAF, based on the CPAB framework, allows to obtain significantly lower approximation error, up to 10 times lower (better) than ReLU, and 3 times better than Tanh. This example further motivates harness CPAB the blueprint of DIGRAF.
21

---

Table 5: Graph classification accuracy (%) 1 TUDatasets. The top three methods marked by
First, Second, Third.
Method D/Daaset
MUTAG
PTC
PROTEINS
NCII
NCT109
STANDARD ACTIVATIONS
GIN + Ideutity
91.4#5.6
66.2+55.5
75.9+3.2
828++20
82.8+11.3
GIN + Sigmoid [69]
90.94:5.5
65.3:14.8
75.0+550
8264114
81.2+:1.6
GIN ReLU [84]
89.415.6
64.6.17.0
76.2.128
82.71177
82.21.16
GIN + LeakyReL.U [50]
90.911..7
65.0:99.0
76.2444
83.522.
82.9
Tanh [35]
92.317.99
83.2.12.6
2.6
GIN + Gel.U [34]
90.944..0
65.4417.9
76.6:12.8
83.5
1.
GIN + ELU [12]
92.5-5.66
65.447..5
75.422.7
83.342.00
82.6+
LFARNABIEE. ACTIVATIONS
66.91:7.0
76.7:13355
82.9:22.6
82.31
GIN + Maxout [29]
91.517.55
66.8.18.3
76.8
83.3:12.9
33.0
GIN + Swish [66]
90.444.8
76,244422
83.:14
82.943.0
GRAPH ACTIVATIONS
[38]
67.7:19.2
75.9:3111
83.3::2.00
82.71:19
GIN + Median [38]
92.0116.6
67.7 14.5
75.0.14.
13.6199
82.8.1.88
+ GRe..U [92]
92.0-27.3
64.946666
76.833
82442.2
GIN + DIGRAF (W/O ADAP.)
92.0115.6
68.9
71.2.1.6
83.0.11.3
82.91222
92.11:799
68.617.4
77.9334
83.4::2
83.31-99
Table 6: Different GNN architectures (GCN, GAT, GIN, SAGE) coupled with ReLU, DIGRAF (W/O ADAP.) and DIGRAF activation functions. The top performing model is marked with the corresponding color for each architecture.
Activation
Model
CORA f
CITESEER
PUBMED
BLOG CATALOG
FLICKR
ZINC J
MOLIIV ?
GCN
79.2=114
67.7:1233
77.61:2.2
12.1::19
50.7:1:.33
0.367410.0111
76.06:10.99
78.0-2.1
7.701:17
74.2.1..
ReI.U
GIN
67.1-300
58.8:22.2
68.4+2277
72.622.5
43.1:226
0.1630=::0.040
75.58-+1.4
SAGE
78.5=1.6
67.4:1..8
76.2+1188
84.943.1
43.5:22.88
0.4680==0.0030
77.66+0991
GCN
81.5=111
69.2::2111
7.631:1.6
80.8:00.6
68.6:11888
0.0083
76.62:1
DIGRAF
78.21..00
1.00115
1..80
GIN
80.6-2.3
67.544422
76.0:::.0
82.1
68.0:11.3
0. 0.13822-000082
19:19:
(W/O ADAP)
SAGE
79.3=7.8
67.742255
77.1#:32
90.6=0.33
66.5+6655
0,4442+
78.19+0.83
GCN 82.8211
69.5:1144
79.3+:1.4
81.62088
69.6:20.6
0.2830-20.0054
77.38:42.31
7..4711118
DIGRAF
GIN 80.6-2.0
68.9113.8
76.91:3.3
83.0&4.11
70.6:14.3
0,1002-1...000
80.28-L: .44
SAGE
79.9-69
68.0:14.0
77.3233
90.8:00.4
(9.0:14.88
0,4147:0.0078
79.32:40.74
E.2
Results on TUDatasets
Our results are summarized Table 5. where we consider MUTAG, PTC, PROTEINS, NCI and NC1109 datasets from the TU repository [56]. As can be seen from the table, DIGRAF is consistently among the top-3 best-performing activation functions, and it consistently outperforms other graph-adaptive activation functions. These results support design choices for DIGRAF and the flexibility offered by CPAB diffcomorphisms.
E.3 Comparison with Different GNN Architectures
We now provide a comparison between DIGRAF and the ReLU activation function coupled with GCN, GAT, GIN, and SAGE backbones in Table 6. Notably, DIGRAF consistently outperforms ReLU regardless of the backbone architecture.
E.4 Visualization of DIGRAF
To gain a qualitative understanding of the behavior of DIGRAF, now illustrate the activation function learned by DIGRAF after the last GNN layer different graphs. To this end, randomly selected two graphs from the ZINC dataset, as shown Figure 7. original graphs are presented the lower right, with each color representing a feature. Nodes with the same color share the same feature. The comparison of the figures demonstrates that for different graphs, with different features and structures. DIGRAF learns distinct activation functions, showing its adaptivity to the input graph.
22

---

Activation Function
Activation Function
0.8-
0.8-
0.6-
0.6
0.4
0.4
0.2
X
X
(a) ZINC Test Graph
(b) ZINC Test Graph 141
Figure 7: Activation function learned by DIGRAF after the last GNN layer two randomly selected graphs from ZINC Different node colors indicate different node features. DIGRAF yields different activations for different graphs.
Table 7: Performance Comparison of DIGRAF with ReLU variants of increased parameter budget. The number of parameters is reported within the parenthesis adjacent to the metric. We use GINE [37] as backbone. Increasing the parameter count with ReLU does not yield significant improvements, and AFGRAF outperforms all variants, even those with higher number of parameters. Note that, DIGRAF (W/O ADAP.) has only N additional parameters, where Np is the tessellation size.
Method / Dataset +
ZINC (MAE L
MOLHIV (Acc. % 1)
GIN + ReLU (standard)
0.1630110.0040 (~ 308k)
1..58:11.40 (~> 63K)
GIN ReLU (double #channels)
0.1578110.0014 ( 1207k)
75.73 (~> 240K)
GIN ReLU (double #layers)
0.1609=000003 (~ 580K)
75.78=+0.43 (~ 116K)
DFGRAF (W/O ADAP.)
0.1382+ (~ 308K)
79.19 (~ 63K)
DIGRAF
0.1302+0.0090 (~ 333K)
80.28 1.44
(~ 83K)
E.5
Parameter Count Comparison
GNN Acr is a core component of DIGRAF, which ensures graph-adaptivity by generating the parameters (()) of the activation function conditioned on the input graph. While the benefits of graph-adaptive activation functions evident from experiments in Section 5, DIGRAF consistently outperforms DIGRAF (W/O ADAP.), not graph adaptive, comes at the cost additional parameters learn GNN,cr (Equation (9)). Specifically, because in all our experiments GNN is composed of 2 layers and a hidden dimension of 64, DIGR AF adds at most approximately 20k additional parameters. The number of added parameters in DIGRAF (W/O ADAP.) is significantly lower, counting at Nyp 1, where Np is the tessellation size. Note in our experiments, the tessellation size does exceed 16. To further understand whether the improved performance AF is due to increased number of parameters, we conduct an additional experiment using the ReLU activation function where we increase the number of parameters of the model and compare the performances. In particular, we consider following settings: (1) The standard variant (GIN ReLU), (2) The variant obtained by doubling the number of layers, and (3) The variant is obtained by doubling the number hidden channels.
variant our method
We present the results of the experiment described above We observed that adding more parameters to the ReLU baseline does not produce significant performance improvements, even in cases where the haselines have <44 times more parameters than DIGR A Fand its baseline. On the contrary, with DIGRAF significantly improved performance obtained compared to the baselines.
ZINC- 12K MOLHIV datasets Table
F Ablation Studies
present impact several key components of DIGRAF, namely the tessellation size Nr. the depth of GNNscr (Equation (9)) and the regularization coefficient À of ((0) (Equation (12)). We choose few
23

---

800
770
(6)
700
200
670
the increased Figure 8: Impact of tessellation size N& on the performance of DIGRAF on CORA, CITESBBR, FLICKR, BLOGCATALOG, and MOLHIV datasets.
Table 8: Effect of depth of GNN ACT on DIGRAF.
Dataset
Face
Locr
FLICKR
69.6110.6
66.3:10.8
69.3
BLOGCATALLOG
81.0:10.5
81.1:1:0488
81.6.20.8
MOLHIV
80.28:11.44 1.44
80.19::1.49
1.55
ZINC
0..0090.0.0000
0.1309+0.0084
0.1314110.083
representative datasets, i.e., CORA, CITESEER, FLICKR and BLOGCATALOG for which we use GCN [43]; and ZINC-12K and MOLHIV for which we use GINE [37] as GNN respectively.
F1
Tessellation Size
Recall that the tessellation size Np determines the dimension of (()) R----1 that parameterizes the velocity fields within DIGRAF. We study the effect of the tessellation size the performance of DIGRAF in Figure 8. can see that small tessellation marginal changes. This observation suggests that CPAB is highly flexible, and aligns with the conclusions in previous studies on different applications of CPAB [52], which have shown that small sizes are sufficient in most cases.
sufficient for good performance,
increasing its size results
F.2 Depth of GNNNCT
DIGRAF exhibits graph adaptivity by predicting (()) E R-Vp-1 conditioned on the input graph through GNNacT- ACT- Table 8 shows the impact of the number of layers Tacr of GNN the performance of DIGR AF. particular, maintain fixed architecture for DIGRAF and vary only Lacr. The results show that increasing depth of GNN.cr improves only marginally number of parameters is not the main factor of the better performance of DIGRAF. On the contrary. the flexibility and adaptivity offered by DIGRAF are the main factors of the improvements, demonstrated by DIGRAF consistently outperforming DIGRAF (W/O ADAP,) and other activation functions (Section 5).
performance DIGRAF, demonstrating
F.3 Regularization
As discussed in Section 4.2, the regularization enforces smoothness the velocity field. investigate impact of the value of the regularization coefficient À on DIGRA AF (Equation (12)) Table 9. results reveal that the optimal value of X depends on the dataset of interest, with small positive values yielding generally good results across all datasets.
F44 Comparison of DIGRAF and DIGRAF (W/O ADAP.) with Equal Parameter Budget
To demonstrate the efficacy of graph adaptivity provided by GNNACT. ACT. conduct experiment where increase number layers and channels of GNNLwvx DIGRAF (W/O ADAP.) to match the total number of parameters in DIGRAF. As shown in Table 10, the increase in the number of parameters does not translate to better performance. Rather, the effective usage of the extra parameters as done by GNNact is the reason behind the performance boost offered DIGRAF.
24

---

Table 9: Effect of velocity field regularization coefficient À DIGRAF.
Dataset
À 0.0
À 0.001
À 0.01
À = 1.0
FLICKR (% ACC
69.1
68. 7.0.7
69.6=0.6
69.0=0099
BLOGCATALOG (% Acc 1)
80,5==0.9
81,0=0.88
81.4=1.0
81,6=0.8
MOLHIV (% ACC
79.38:22.10
80,16:1.50 0.1348.1000093
80.28 1.44
1.21
ZINC (MAE J)
0.1395.L0.0102
0.1002.10.0000
0.1353
Table 10: Results on ZINC and MOLHIV datasets along with number of parameters in parenthesis.
Method
ZINC (MAE) 4
MOLHIV (ROC AUC)
DIGRAF (W/ ADAP. with larger GNNuayer
0,1388 0.0071 (337K)
79,22 1.40 (85K)
GIN DIGRAF (W/ ADAP.) (Original)
0,1382 0.0086 (308K)
79.19 + 1.36 (63K)
GIN DIGRAF
0,1302 0.0094 (333K)
80.28 1.44 (83K)
G Experimental Details
We implemented DIGRAF using PyTorch [63] (offered under BSD-3 Clause license) and the PyTorch Geometric library [22] (offered under MIT license). All experiments were conducted on NVIDIA RTX A5000, NVIDIA GeForee RTX 4090, NVIDIA GeForee RTX 4070 Ti Super, NVIDIA GeForce GTX 1080 Ti, NVIDIA TITAN RTX and NVIDIA TITAN V GPUs. For hyperparameter tuning and model selection, we utilized the Weights and Biases (wandb) library [6]. We used the difv package [52, 25, 24, 74] (offered under MIT license) for the diffeomorphic transformations based on the closed-form integration of CPA velocity functions. In the following subsections, we present the experimental procedure, dataset details, and hyperparameter configurations for each task.
The hyperparameters include the number of layers L and embedding dimension C' of GNN AATER learning rates and weight decay factors for both GNN and GNNacT, dropout rate p, tessellation size Np. and regularization coefficient A. We additionally include the number of layers Laer and embedding dimension Cscr of GNN.cc. We employed combination of grid search and Bayesian optimization. All hyperparameters were chosen according to the best validation metric. For the baselines, we include only the applicable hyperparameters in our search space.
Hyperparameters.
Node Classification. For each datasct, we train a 2-layer GCN [43] as the backbonc architecture, and integrate each of the activation functions into this model. Following Zhang et al. [92]. we randomly choose 20 nodes from each class for training and select 1000 nodes for testing. For each activation function, run the experiment 10 times with random partitions. We report the mean and standard deviation of node classification accuracy on the test set, Table 11 summarizes the statistics of the node classification datasets used our experiments. All models were trained for 1000 epochs with a fixed batch size of 32 using the Adam optimizer. Tables 12 and 13 lists the hyperparameters and their search ranges or values.
Graph Classification. setup:
The statistics of various datasets can be found in Table 14. We consider the following
We consider the splits provided in Dwivedi et al. [17]. We use the mean absolute error (MAE) both as the loss and evaluation metric and report the mean and standard deviation over the test set calculated using five different seeds. We use the Adam optimizer and decay the learning rate by 0.5 every 300 epochs, with a maximum of 1000 epochs. In all our experiments, we adhere the 500K parameter budget [17]. We use GINE [37] layers both for GNN!)) and we fix CA 64 and Laci 2. We report the hyperparameter search space for all the other hyperparameters in Table 15.
ZINC-12K:
LAvER and within GNN
TUDatasets: We follow the standard procedure preseribed in Xu et al. [84] for evaluation. That is, we use a 10 fold cross-validatin and report the mean and standard deviation of the accuracy at the epach that yields the hest validation performance on average. We use the Adam optimizer and train for a maximum of 350 epochs. We use GIN [84] layers both for G and within GNN AcT. and we fix LACT 2. We present the hyperparameter search space for all other parameters in Table 15.
OGB: We consider 4 datasets from the OGB repository, with one, namely MOLESOI, being a regression problem, while the others are classification tasks. We run cach experiment using five different seeds and report the mean and standard deviation of RMSE/ROC-AUC. We use the Adam optimizer, decaying the learning rate by factor of 0.5 every 100 epochs, and train for maximum of 500 epochs. We use the GINE model with the encoders prescribed in Hu et al. [36] both for LAYER (N
25

---

Table 11: Statistics of the node classification datasets [53, 73, 58, 86].
Dataset
#nodes
Meriges
#features
#cllases
PLANETOID
CORA
2,708
10,556
1,433
CITESBER
3,327
9,104
3,703
PURMED
19,717
88,648
500
SOCIAL NETWORKS
FLICKR
7,575
479,476
12,047
9
BLOGCATALOG
5,196
343,486
Table 12: Hyperparameter confi for the Planetoid datasets [53, 73, 58).
Hyperparameeter
Search Range/ Value
Learning rate GNYE
[10 5
10
10
5
9,5 x 10
Learning rate g(t) /GNNcc
[10
0,5
6
10
10 10 8,5 x 10
Weight docny
[10
10
5 10 9,0.0]
[61. 128, 256]
Cur LACT P Np
128]
[2. 4] [0.0, 0.5] [2. 12.4.8.
A
[0.0, 10 1.000-2
100..00- 1..00.100
and within GNNACT, and we set CACT 64 and Laer 2. We present the hyperparameter search space for all other parameters in Table 15
H Complexity and Runtimes
now provide an analysis of the time complexity DIGRAF. Let us recall the following details: (i) As described in Equation (8), DIGRAF is applicd element- wise paraliel for each dimension of the output GNN(") (ii) As described in Equation (9), we employ an additional GNN denoted LAVIR
Time Complexity.
by GNN to compute (()) In all our experiments, both the backbonc GNN and GNN AcT peessage-passinggg neural networks (MPNNs) [28]. (iii) As described in Theorem of Freifeld al. [25], for I-dimensional domain, there exists a closed form for (:; ())), and the complexity for the CPAB computations are linear with respect to the tesselation size, which is constant of up to 16 in our experiments. Therefore, using DIGR AF with any linear complexity (with respect to the number of nodes and edges) MPNN-based hackhone maintains the linear complexity of the backbone MPNN. Put precisely, each MPNN layer has linear complexity in the number nodes |V] and [E'. We use Laer layers GNN the computational complexity of DIGRAF layer is O (LaCr [[V] + (E)). Since we have L layers in overall GNN, the computational complexity of
an MPNN-based GNN coupled with AIGRAF is O(L LAcr (VVI + [E))). In our experiments, we fix the hyperparameter Lxcr 2, resulting in O(L (IV1 [E])) computational complexity in practice.
Memory Complexity. DIGRAF uses GNN which is an MPNN and hence has linear space complexity (with respect the number of nodes and edges). CPAB computations require constant memory with respect to the graph size I-dimensional domain due to analytical implementation. We use layers in overall GNN and Lacr layers in GNNacr resulting in a memory complexity of OLL LICT - ([V) + [E)). In our experiments, we fix the hyperparameter Lxer = 2, resulting in OLL ([V) + [B)) memory complexity in practice.
Runtimes. Despite having linear computational complexity in the size of the graph, DIGRAF performs additional computations to obtain (²) using GNN To understand the impact these computations, we measured the training and inference times of DIGRAF and present it in Table 16. Specifically, we report the average time per batch and standard deviation of the same measured on NVIDIA A5000 GPU, using a batch size of 128. For fair comparison, Additionally, for our DIGRAF, we set the number of layers within GNN dimension to CACT = 64. Our analysis indicates that while DIGRAN AF requires additional computational time, yields significantly better performance. For example, compared to the best activation function on the dataset, namely Maxout, DIGRAP requires an additional ~ 6.21ms at inference, but results in a relative improvement in the performance of 17.95%.
the same number of layers, batch size, and channels in all methods. Lxer 2, and the embedding
On the ZINC dataset, using GIN as the primary model, DIGRAF exhibits approximately 4.5 times slower training times 3.5 times slower inference times compared to ReLU, DIGRAF demonstrates an inference time that is approximately 1.35 times faster than GReLU, while also achieving superior performance.
26

---

Table 13: Hyperparameter configurations for the social network datasets [86].
Hyperparameker
Scarch Range/ Value
Learning rate for GNN
[10
10 5 x 10
10 3,5 10-2, 10 10
Leariing rale for g(9) /GNNAY
[1079
10
10
10 10-2,. 10
Weight deeny for GNN, srns
[10-9,
10 xx 10 10.0
Weight decay ((0) AGNNAeT
[10-6,
10-0 10 1- 107 .0.0)
C
664, 128, 256
CARY
[16, 32, 128] 2.4 [2,4] [0.0, 0.4, 0.6.6. [2. 4, 8, 16
L
Lacr
Np
[0.0, 10 ,100 11002,100) 1.0]
Table 14: Statistics of the graph classification datasets [55, 36, 17].
Datieet
#graphs
#nodes
#edges
Hecature
Velasses
ZINC-12K
12,000
223.
^.49,8
T'Daaaases
MUTAG
188
17.9
</99
7
PROTEINS
1,113
~v39.1
14456
PTC
344
14.2
14.6
18
NCTI
4,110
2229.
~-22.
37
NCIL09
4,127
2929
3-2.1
OGB
MOLESOL
1,128
1=33
13.77
MOLTON21
7,831
~188
199.
MOLBACE
1,513
3441
~369
MOLHIV
41.127
2255
2/275
9
Table 15: Hyperparameters and search ranges/values for TUDatasets [55], OGB [36], and ZINC- 12K [17] datasets.
Hyperparaneter
TUDalasels
OGB
ZINC
Learning GNN
Lavin
[10-5
10-4
11000
10
Learning for (()) JGNNs
15 10-6
10
100
10
10
Weight docay for GNN (2) Weight decny (f JGNN
[10
10
0.00]
110
4.5
8.0.0]
C'
[16, 32]
664, 128]
[61, 128,
CAeY
116, 32, 64, 128]
L
[4,
[2
[2.4]
0.5
AP
8,
[0.0, 10-3. 10 2.0
Graph pooling layer Batch
[sum, mean]
128]
128]
[64, 128]
Table 16: Batch runtimes on NVIDIA RTX A5000 GPU of DIGRAF and other activation functions, with GNN layers, batch size 128, 64 embedding dimensions, and GNN with L = 2 layers and CACr 64 embedding dimension, ZINC- 12k dataset.
ZINC Inference time
Method
Training time (ms)
(MAE
Rel.U
4.18-00100
2.47.10.08
0.1630-10.0040
GIN Maxout [29]
4.71-0.13
2.110..12
0,1587+000007
GIN [66]
4.55=0.12
2.30=:0.24
0.1636:0.0009
Max [38]
9.19=0.25
4.50-0..3
0.0661+0.0005
GIN Median [38]
14:54-:11.5
10.13=1.20
0.1715=00.000
GIN GReLU [92]
20.63-10.99
11.69=2.779
0,3003=40.0086
GIN DIGRAF ( (W/ ADAP.
13.76.10.65
4.97.L
0.0080000.0080
DIGRAF
19.3741.22
8.62.40.18
0.3302+0.0000
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Section and Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Section 6 and Appendix H.
Guidelincs:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are violations of
these assumptions (e.g.. independence assumptions, noiseless settings, model well- spec--iifccitnon asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures hecause fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If l'applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 'te't acknowledged in the paper. The authors should that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
their best judgment and recognize
Theory Assumptions and Proofs
Question: For each theoretical result. does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We introduce the idea in Section 4.3, with additional propositions and detailed proof' show in Appendix D.
Guidelincs:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
28

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
Formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 5, Appendix E, Appendix B and Appendix G.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice,
example, if the contribution or the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the of large language model), releasing of model checkpoint, or other means that are appropriate to research performed.
While NeurIPS does not require relcasing code, the conference does require submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a way to access this model for reproducing the results a way to reproduce model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
new model (e.g.. large language model), then there should either be
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to deseribe the particular way they provide closcd-sourc source models, it may be that access to the model is limited some way (c.g., to registered users), but it should be possible for other rescarchers to have some path to reproducing or verifying the results.
reprodueibilityy In the case of
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described supplemental material?
Answer:
[Yes]
Justification: We provide extensive details on the implementation and evaluation of method, and we released our code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurTPS code and data submission guidelines http:://iipsccccpbb/be quides/CodeSubae
ieeinnooiccyy for more details.
While we encourage release of code and data, we understand that this might not possible,
so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source henchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: :/nips....///// cc/public/ Tua.des/CCoce@ubbmissssoo/oool for more details.
29

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: See Appendix G and Scction 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: The standard deviations for all the metrics have been presented. See Table 1, Table 2, Table 3, Table 5, Table 7, Table 8, Table 9, and Table 16. Also see Section 5, Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some paramcter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma enror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, authors should be careful not to show in tables or figures
symmetric error hars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment. does the paper provide sufficient information the computer resources (type of compute workers, memory, time exccution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We discuss the computational resources Appendix G and Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
30

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, every respect, with NeurIPS Code of Ethics https:/ httpss//eeurips..../ cc/ httpppullc/tthhaauuuuiieee
Answer: [Yes]
Justification: See Appendix G.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
authors should make sure preserve anonymity (e-g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impact in the conclusion, see Section 6.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will he foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example. it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We don't 't scrape any datasets not we release any models of high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example hy requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
31

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited each datasets we used in Section 5 and Appendix G. The license of the packages we used is in Appendix G.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are relcased, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ catasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that re-packaged. both the original license and the license of the derived asset (if it changed) should provided.
this information not available online, the authors encouraged reach out the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We don't ton't release any new assets.
Guidelines:
The NA means that the paper does not release
Researchers should communicate the details of the dataset/code/mmodel as part of their sub- missions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least minimum wage in country data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
32

---

Answer:
[NA]
Justification: involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, approval (or equivalent) may be required for any human subjects research. If you ohtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
33


Reviews:
[{"review_id": "MHsRjLGP69", "reviewer": "Reviewer_mZzM", "summary": "The paper “DiGRAF: Diffeomorphic Graph Activation Functions” introduces DiGRAF, a novel graph neural network (GNN) activation function based on diffeomorphisms. DiGRAF adapts to graph structures and tasks by learning transformation parameters, enhancing performance across various GNN scenarios. The method demonstrates its effectiveness through extensive experiments on node classification, graph classification, and regression tasks, showing significant improvements over standard and graph-specific activation functions.", "questions": "How are the parameters {b_j} (orthonormal basis of the space of velocity fields V) calculated?\n2.\tWhat would be the activation function of GNN_act if the method’s purpose was to design a new activation function?\n3.\tAre there results for other GNN structures beyond GCN? For instance, GReLU reports results for different GNN structures.\n4.\tSince DiGRAF is also applicable to non-graph datasets, would it be beneficial to see its results on other machine learning methods?\n5.\tWhat is the closed-form solution of the integral in Equation (2) for 1-dimensional vectors?\n6.\tIs there any information on the learned \\theta parameters in terms of sparsity or graph structure?", "limitations": "Inductive Learning: Since the method is task-dependent, it loses its inductive capabilities and cannot be used for cases it did not see.\n2.\tHyperparameter Choice: There is a dependency on hyperparameter tuning, which may affect generalizability.\n3.\tTime Complexity: DiGRAF has higher time complexity compared to standard activation functions.\n4.\tDecentralized Tasks: The approach is not ideal for decentralized tasks where there is no central data.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tPower of DiGRAF: The paper effectively demonstrates the superiority of DiGRAF in various GNN scenarios, showcasing its adaptability and performance improvements.\n2.\tExtensive Experiment Analysis: The experiments are comprehensive, covering multiple datasets and comparing them with various baseline activation functions.\n3.\tLearnable Activation Function: DiGRAF uses a flexible, learnable activation function that is differentiable and bounded, providing desirable characteristics for machine learning applications.", "weaknesses": "1.      Limited Contribution: The paper appears to be more of an application of CPAB. The primary contribution of the analysis of the connection between CPAB and GNNs is not thoroughly explored.\n2.\tPerformance and Gradient Optimization: The paper lacks a detailed analysis of performance and gradient optimization aspects.\n3.\tClarification on GNN_act: It needs to be clearer how GNN_act helps boost performance. Is there a way to analyze the new source of information it brings?\n4.\tProofs for GNN: Most analyses focus on the boundedness of the activation function without specifics tailored to GNNs, making it unclear why this approach is particularly suited for GNNs.\n5.\tEdge Classification Performance: The report on edge classification performance is missing.\n6.  \tHyper-Parameter Effects: It is not clear what the effect of each hyper-parameter is on the performance of the activation function.\n7.   \tSource Code: The source code is not available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the insightful comments, and appreciation of our experiment analysis. We now address them. New results and discussions were added to our paper.  \n\n**1. On Contribution**: We distinguish between CPAB's original purpose (signal alignment via diffeomorphism) and our novel use for learning an activation function (Section 4.1 and Figure 4). Additionally, we made several key contributions: (i) Sections 4.2 and 5 demonstrate the importance of correctly parameterizing the velocity field for graph adaptivity, (ii) Section 4.3 discusses DiGRAF's properties, justifying its design and effectiveness as a graph activation function. \n\n**2.On Optimization**: DiGRAF's behavior is analyzed by its Lipschitz constant (Proposition D.2), related to its optimization[5,6]. Figure 5 and Appendix E.1 show DiGRAF's faster convergence and lower loss compared to other activation functions. \n\n**3.On $\\text{GNN}_\\text{act}$**:  $\\text{GNN}_\\text{act}$ adapts the activation function to the graph, by returning the parameter $\\theta$ for the learned diffeomorphism. This graph adaptivity is crucial, as shown by DiGRAF consistently outperforming its non-adaptive counterpart (DiGRAF w/o ADAP).\n\nFigure 2 in the additional PDF shows learned activation functions for two graphs from ZINC, differing in number of nodes, features, and connectivity. $\\text{GNN}_\\text{act}$ enables DiGRAF to capture these differences and learn distinct activation functions for them.\n\n**4.On DiGRAF Analyses**: DiGRAF is well-suited for GNNs because $\\theta$, determining the activation function (Equation 7), is learned by $\\text{GNN}_{\\text{act}}$ (Equation 9), making it adaptive to different input graphs. Our experiments and response 3 highlight that adaptivity enhances performance. \n\n**5.On Edge Classification**:  Following your suggestion, we performed experiments on link prediction using GCN+DiGRAF on the OGBL-Collab dataset (**Table 2, Additional PDF**), showing that DiGRAF improves ReLU and surpasses DiGRAF (w/o ADAP) on this additional task, supporting our claims of consistent improvements.\n\n**6.On Hyper-parameters**:  We provide experiments on key hyper-parameters (Appendix F). A small tessellation size suffices for good performance (Figure 7). Increasing $\\text{GNN}_\\text{act}$ depth marginally improves DiGRAF's performance (Table 6). Regularization coefficient $\\lambda$ varies by dataset, with small positive values yielding good results (Table 7). \n\n**7. On Code**: As promised in our submission, we will publicly release the code upon acceptance.\n\n**8. On $b_j$**:The orthonormal basis $\\mathbf{B}$ for the velocity field is obtained via SVD of $\\mathbf{L}$, which constrains velocity function coefficients by ensuring consistent values at shared endpoints [4].\n\n**9. On Activation in $\\text{GNN}_\\text{act}$**: We use ReLU within $\\text{GNN}_\\text{act}$, as it is  widely used in GNNs, to make it non-linear. \n\n**10. On GNN Structures**: Our experiments include GCN and GIN. With your suggestion, we additionally tested GAT [1] and SAGE [2]. Table 1 in the additional PDF shows that DiGRAF continues to consistently offer superior results. \n\n**11. On Non-graph data**: Graphs are more diverse than other data due to their unstructured nature, making adaptativity more crucial. Still, DiGRAF can potentially be specialized to other domains, like vision, by using a CNN instead of $\\text{GNN}_\\text{act}$ or our non-adaptive variant. We are eager to explore this in future work.\n\n**12. On Closed-form solution**: Eq. (2) has an equivalent ODE [4]. By varying $x$ and fixing $t$ the solution to this ODE can be written as a composition of a finite number of solutions:\n$\\\\phi^{\\theta} (x, t) = (\\psi^{t_m}\\_{\\theta, c_m} \\circ \\psi^{t_{m-1}}\\_{\\theta, c_{m-1}} \\circ \\cdots \\circ \\psi^{t_2}\\_{\\theta, c_2} \\circ \\psi^{t_1}_{\\theta, c_1})(x)$\n\nHere $m$ is the number of cells visited. Given $x, \\theta$, time $t$, and the smallest cell index containing $x$, $c$, we can compute each $\\psi^{t_i}\\_{\\theta, c_i}(x), i \\in \\{1, …, m\\}$ from $\\psi^{t_1}\\_{\\theta,c_1}(x)$ to $\\psi^{t_m}_{\\theta,c_m}(x)$.\nThis iterations continue until convergence, with an upper bound for $m$ being $max(c_1, N_P-c_1+1)$, where $c_1$ is to the first visited cell index, and $N_P$ is the number of closed intervals in $\\Omega$. Unrolling these steps, we obtain the closed form solution for Eq. (2).\n\n**13. On $\\theta$**: $\\theta$ governs the learned velocity field for the activation function. While CPAB transformations don’t require $\\theta$ to be sparse, the sparsest vector (zeros) results in an identity activation function. Predicted by $\\text{GNN}_\\text{act}$, $\\theta$ reflects the graph structure and can produce different functions for different graphs, as discussed in response 3.\n\n**14. On Inductive Learning**: DiGRAF is inductive in the sense of graph learning, it is capable of handling new graphs in tests. By design, it is task and data-driven, which may limit generalization to entirely new tasks. This is common to almost all GNN models, and not specific to DiGRAF. Recent Graph Foundation Models study generalization to new tasks, and it would be interesting to adapt DiGRAF to such models.\n\n**15. On Hyperparameter Tuning**: Tuning is common in GNNs [3]. Our experiments show DiGRAF's effectiveness across diverse tasks and datasets. The ablation study (Appendix F) demonstrates that DiGRAF performs consistently in different hyperparameters.\n\n**16. On Complexity**: While DiGRAF requires more computations than ReLU, its asymptotic complexity is still linear (Section 4.3). Table 13 shows DiGRAF is ~3.5x slower than ReLU but offers significantly better performance. Additionally, DiGRAF is 1.35x faster than other graph activation functions and achieves better performance.\n\n**17. On Decentralized Tasks**: This limitation is common to most GNNs, similar to the discussion in 14. While it is an interesting research direction, addressing it is beyond the scope of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iIZO9ytVyh", "reviewer": "Reviewer_LFcv", "summary": "This paper introduces DIGRAF, Diffeomorphic Graph-Adaptive Activation Function, which is a novel activation function adaptive to graph data. DIGRAF leverages Continuous Piecewise-Affine Based transformations, possesses several necessary properties of a good activation function, such as differentiability, zero-centering and permutation equivariance. The author allows DIGRAF adapt to graph data by using an additional GNN to learn the final activation function formula. Adequate experiments which take use of DIGRAF in GNNs proved the high efficiency of this activation function.", "questions": "1. In the experiments, why was the sigmoid function not used as an activation function?\n2. Are there any differences in the DIGRAF learning process for GCN and GIN?", "limitations": "1. As the author mentioned, the current DIGRAF may not be optimal since it was learned only for simple tasks, but it can be improved in future work.\n2. The application of DIGRAF has not been discussed or implemented in this research. However, using DIGRAF as a novel activation function in some current models might improve their efficiency.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe activation function DIGRAF is a good choice for GNN model design, more effective than the common activation function such as ReLU. \n2.\tThe soundness of this paper is wonderful. The theoretical proof of DIGRAF’s properties is detailed and experiments in this paper is convictive, which contain node classification, graph classification and graph regression in task level.", "weaknesses": "There should be an ablation study (can be set in the supplementary part) about the hyper-parameters and training strategy in the process of leaning DIGRAF.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad to see the reviewer has particularly appreciated the soundness of our paper, while finding the theoretical analysis detailed and the experiments convincing. We proceed by answering the questions raised by the reviewer in the following. We hope that you find our responses satisfactory to consider revising your score. We are welcoming any question, comment, or suggestion.\n\n\n**Q1**: *There should be an ablation study (can be set in the supplementary part) about the hyper-parameters and training strategy in the process of learning DiGRAF.*\n\n**A1**: We would like to kindly note that Appendix F contains an ablation study on the effect of the hyper-parameters added by DiGRAF. We found that a small tessellation size is sufficient for good performance, and increasing its size results in only marginal changes (Figure 7). Moreover, increasing the depth of $\\text{GNN}_\\text{act}$ improves the performance of DiGRAF marginally (Table 6). For regularization, the results (Table 7) reveal that the optimal value of the coefficient depends on the dataset of interest, with small positive values generally yielding good results across all datasets. Also, we discuss the training loss, protocols and hyperparameter tuning in Appendices B and G, respectively. In our revised paper, we improved the link between the relevant parts in the main paper and the appendices. Thank you.\n\n**Q2**: *In the experiments, why was the Sigmoid function not used as an activation function?*\n\n**A2**: We appreciate the reviewer's observation. We have conducted additional experiments using the Sigmoid function, and added them to our revised paper. As can be seen from the table below, DiGRAF consistently outperforms Sigmoid with a large margin across various datasets:\n\n|Activation\\Dataset| Blog Catalog $\\uparrow$  | Flickr $\\uparrow$ | CiteSeer $\\uparrow$ | Cora $\\uparrow$ | PubMed $\\uparrow$ |\n|-|-|-|-|-|-|\n| Sigmoid |39.7 $\\pm$ 4.5| 18.3 $\\pm$ 1.2| 27.9 $\\pm$ 2.1 | 32.1 $\\pm$ 2.3 | 52.8 $\\pm$ 6.6 |\n| DiGRAF (w/o ADAP.) | 80.8 $\\pm$ 0.6 | 68.6 $\\pm$ 1.8 | 69.2 $\\pm$ 2.1 | 81.5 $\\pm$ 1.1 | 78.3 $\\pm$ 1.6 |\n| DiGRAF | 81.6 $\\pm$ 0.8 | 69.6 $\\pm$ 0.6 | 69.5 $\\pm$ 1.4 | 82.8 $\\pm$ 1.1 | 79.3 $\\pm$ 1.4|\n\n\n|Activation\\Dataset|  MUTAG $\\uparrow$  | PTC $\\uparrow$ | PROTEINS $\\uparrow$ | NCI1 $\\uparrow$ |  NCI109 $\\uparrow$ |\n|-|-|-|-|-|-|\n| Sigmoid | 90.9 $\\pm$ 5.5 | 65.3 $\\pm$ 4.8 | 75.0 $\\pm$ 5.0 | 82.6 $\\pm$ 1.4 | 81.2 $\\pm$ 1.6 |\n| DiGRAF (w/o ADAP.) | 92.0 $\\pm$ 5.6 | 68.9 $\\pm$ 7.5 | 77.2 $\\pm$ 3.6 | 83.0 $\\pm$ 1.3 | 82.9 $\\pm$ 2.2 |\n| DiGRAF | 92.1 $\\pm$ 7.9 | 68.6 $\\pm$ 7.4 | 77.9 $\\pm$ 3.4 | 83.4 $\\pm$ 1.2 | 83.3 $\\pm$ 1.9 |\n\n\n|Activation\\Dataset| MOLESOL (RMSE) $\\downarrow$| MOLTOX21 (ROC AUC) $\\uparrow$ | MOLBACE (ROC AUC) $\\uparrow$ | MOLHIV (ROC AUC) $\\uparrow$ |\n|-|-|-|-|-|\n| Sigmoid | $0.8836 \\pm 0.043$ | $69.15 \\pm 0.52$ | $68.70 \\pm 3.68$ | $73.87 \\pm 0.80$ |\n| DiGRAF (w/o ADAP.) | $0.9011 \\pm 0.047$ | $76.37 \\pm 0.49$ | $78.90 \\pm 1.41$ | $79.19 \\pm 1.36$ |\n| DiGRAF | $0.8196 \\pm 0.051$ | $77.03 \\pm 0.59$ | $80.37 \\pm 1.37$ | $80.28 \\pm 1.44$ |\n\n\n|Activation\\Dataset|  ZINC (MAE) $\\downarrow$  |\n|-|-|\n| Sigmoid | $0.3839 \\pm 0.0058$ |\n| DiGRAF (w/o ADAP) | $0.1382 \\pm 0.0080$ |\n| DiGRAF | $0.1302 \\pm 0.0090$ |\n\n\n\n**Q3**: *Are there any differences in the DiGRAF learning process for GCN and GIN?*\n\n**A3**: There are no differences in the learning process for DiGRAF in GCN and GIN. Specifically, although the GNN layer in $\\text{GNN}\\_{act}$ aligns with the primary GNN model, $\\text{GNN}\\_{act}$ still takes in the input graph data and returns the parameter $\\theta$ that is used in the activation DiGRAF. In terms of training, we did not change the training procedure for different backbones. We added this important discussion to the revised paper. Thank you.\n\n**Q4**: As the author mentioned, the current DIGRAF may not be optimal since it was learned only for simple tasks, but it can be improved in future work.\n\n**A4**: We thank the reviewer for their comments. While we evaluated DiGRAF on a wide variety of available real-world datasets and tasks, which are widely utilized by the GNN community, it would be interesting to see DiGRAF performance on additional real-world tasks, and we are eager to explore this in future work. Our motivation in demonstrating DiGRAF on such a variety of datasets and benchmarks is to show its general effectiveness, and its ability to cater different communities that utilize GNNs to solve hard problems like drug discovery and weather prediction. Overall, we found that DiGRAF consistently offers better performance than other activation functions, thereby motivating its application in additional tasks. We thank you for the insightful comment, and we added this discussion to our revised paper.\n\n**Q5**: *The application of DiGRAF has not been discussed or implemented in this research. However, using DiGRAF as a novel activation function in some current models might improve their efficiency.*\n\n**A5**: We believe the reviewer is referring to applying an adaptive activation function, such as DiGRAF, to other domains. While this represents an intriguing avenue of research, it falls outside the scope of the current work, which specifically focuses on graph inputs, as discussed throughout our paper. DiGRAF leverages graph information, making it particularly well-suited for graph-related tasks. We believe however specialization to other domains is possible and we are happy to try in future work. It is also important for us to note that to comprehensively demonstrate the effectiveness of DiGRAF, our experimental section was carefully designed to include different tasks, from node to graph level, inductive and transductive, spanning across various domains, from citation networks to molecular datasets. We further highlighted these aspects in our revised paper - thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gd0TH52wWg", "reviewer": "Reviewer_SUte", "summary": "This paper introduces a new activation function, DIGRAF, specifically tailored for graph data in Graph Neural Networks (GNNs). The approach is based on Continuous Piecewise-Affine Based transformation (CPAB). The authors demonstrate that DIGRAF possesses the desired properties highlighted in existing literature and provide a thorough analysis of these properties. Extensive experiments conducted on diverse datasets across various tasks show that DIGRAF outperforms three different types of baselines in downstream performance.", "questions": "1. The size of GNN_act seems to have a minor influence (non-monotonic) on performance according to Appendix F.2. What if you increase the size of GNN_layer a little bit (add the parameter of GNN_act to GNN_layer) and compare DIGRAF (w/o adap. & larger GNN_layer) with the current DIGRAF? I believe this could provide a better justification for the effectiveness of GNN_act. \n2. What are the properties that DIGRAF has but general-purpose activation functions or existing graph activation functions do not have? Can you put up a table to clarify it?\n3. How do the properties of DIGRAF influence the convergence and performance? \n4. The tessellation size decides the degree of freedom of DIGRAF, right?  Why does increasing the tessellation size result in marginal changes (non-monotonic)?", "limitations": "The authors have discussed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper devise an activation function based on CPAB, which possess many desirable properties of activation functions (e.g. zero-centered, permutation equivariant, etc.) and has solid theoretical guarantees. It's an ingenious idea and seems that this method has never been adopted before in the activation function of GNN.\n2. The variables and equations are well-defined and thoroughly explained. The experiments are extensive and robust, effectively supporting the corresponding claims with comprehensive data.\n3. The paper addresses the most critical questions about the effectiveness of the proposed method. Each question is answered with extensive experiments and explicit explanations, providing clear evidence for the method’s efficacy. The figures are well-designed, enhancing understanding and readability. Overall, the paper is smoothly written and easy to follow.", "weaknesses": "1. Apart from the adoption of the CPAB approach, the primary contribution appears to be the introduction of GNN_act. However, the discussion on why GNN_act is effective is relatively brief. Including more detailed discussions or theoretical justifications would help in understanding its advantages.\n2. How do the properties of DIGRAF influence the convergence? It would be nice if it could be discussed which property the gain comes from.\n3. As I mentioned above, more ablation studies should be done to reveal how each property and design contribute to the performance gain. Otherwise, DIGRAF (w/o adap.) is good and simple enough. Maybe we can save the budget of GNN_act for a larger GNN_layer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer’s constructive and positive feedback. We are delighted to see the reviewer has valued our experimental analysis. We proceed by answering the questions in the following.\n\n**Q1**: *The discussion on why $\\text{GNN}_\\text{act}$ is effective is relatively brief. Including more detailed discussions or theoretical justifications would help in understanding its advantages.*\n\n**A1**: We thank you for your suggestion, and mention that $\\text{GNN}\\_\\text{act}$ boosts performance by adapting the activation function to the input graph, specifically by returning the parameter $\\theta$, which governs the CPAB diffeomorphism used as an activation function. This graph adaptivity represents the critical factor of the performance boost, as evident by DiGRAF consistently outperforming DiGRAF (w/o ADAP), where $\\text{GNN}\\_\\text{act}$ is replaced by a learnable $\\theta$ which is the same for all graphs.\n\nTo make this point clearer, in **Figure 2 in the additional PDF**, we plot the learned activation functions for two distinct randomly sampled graphs from the ZINC dataset, which differ in the number of nodes, features, and the connectivity. While using the non-adaptive variant DiGRAF (w/o ADAP) yields a (learned) fixed activation function for all inputs,  $\\text{GNN}_\\text{act}$ can account for the variables discussed above, and  learn distinct activation functions for different graphs.\n\n**Q2**: *The size of $\\text{GNN}\\_\\text{act}$ seems to have a minor influence (non-monotonic) on performance according to Appendix F.2. What if you increase the size of $\\text{GNN}\\_{\\text{layer}}$ a little bit (add the parameter of $\\text{GNN}\\_\\text{act}$ to $\\text{GNN}\\_{\\text{layer}}$) and compare DiGRAF(w/o ADAP. \\& larger $\\text{GNN}\\_{\\text{layer}}$ ) with the current DiGRAF?*\n\n**A2**: As shown in Table 5 in Appendix E, for ZINC and MOLHIV datasets the difference in the number of parameters between DiGRAF and DiGRAF (w/o ADAP) is relatively small, with DiGRAF having 10% and 30% more parameters than DiGRAF (w/o ADAP), respectively. However, we welcome the reviewer’s suggestion and we have now conducted an additional experiment, where we increase the number of parameters of DiGRAF (w/o ADAP) to match the one of DiGRAF by increasing the size of $\\text{GNN}_\\text{layer}$. We denote this variant by DiGRAF(w/o ADAP. \\& larger $\\text{GNN}\\_{\\text{layer}}$ ).\n\nIn the Table below, we report the performance on the ZINC-12k and OGBG-MOLHIV datasets. The results show that the improvement of DiGRAF cannot be attributed to the (small) increase in the number of parameters. On the contrary, it is the way these parameters are allocated, namely for $\\text{GNN}_\\text{act}$, which adapts the activation function to the graph, that yields significant improvements.  \n\n| Method  | ZINC (MAE) $\\downarrow$  | MOLHIV (ROC AUC) $\\uparrow$|\n|:--|:--|:--:|\n| DiGRAF (w/o ADAP.) with larger $\\text{GNN}_\\text{Layer}$ | 0.1388 $\\pm$ 0.0071 (337K)   | 79.22 $\\pm$ 1.40 (85K)  |\n| DiGRAF (w/o ADAP.) (Original)  | 0.1382 $\\pm$ 0.0086 (308K)   | 79.19 $\\pm$ 1.36 (63K)  |\n| DiGRAF   | 0.1302 $\\pm$ 0.0094 (333K)   | 80.28 $\\pm$ 1.44 (83K)  |\n\nFurthermore, we would like to kindly refer you to a similar experiment in our submission. In Table 5, we show that the contribution of DiGRAF (for all variants) stems from the learning of the activation function, rather than adding more parameters to a baseline model.\n\n**Q3**: *What are the properties that DiGRAF has but general-purpose activation functions or existing graph activation functions do not have? Can you put up a table to clarify it?*\n\n**A3**: We appreciate the reviewer's suggestion. We have identified the desirable properties of activation functions and examined various activation functions in the following table.\n\n| Properties \\ Act | ReLU | Tanh | PReLU | Swish | Max | Median | GReLU | DiGRAF |\n|-|-|-|-|-|-|-|-|-|\n| Boundness | N | Y | N | N | N | N | N | Y (within $\\Omega)$ |\n| Differentiability Everywhere | N | Y | N | Y | N | N | N | Y |\n| Linear Complexity | Y | Y | Y | Y | Y | Y* | Y | Y |\n| Permutation Equivariance | Y | Y | Y | Y | Y | Y | Y | Y |\n| Lipschitz Continuity | Y | Y | Y | Y | Y | NA in their paper | NA in their paper | Y |\n| Graph Adaptivity | N | N | N | N | Y | Y | Y | Y |\n\n*In practice, the Median-of-medians Algorithm can achieve linear time complexity on average.\n\nWe discuss these properties in Section 4.3, with proofs of Lipschitz continuity and permutation equivariance provided in Appendix D. We added the Table and a discussion in our revised paper.\n\n**Q4**: *How do the properties of DIGRAF influence the convergence and performance?*\n\n**A4**: DiGRAF possesses the key properties typically associated with faster convergence: differentiability everywhere, boundedness within the input-output domain $\\Omega$, zero-centrality (Szandala, 2021, Dubey et al., 2022). In our revised paper, we improved the connection between these properties and the faster convergence and performance offered by DiGRAF. \n\n**Q5**: *The tessellation size decides the degree of freedom of DiGRAF right? Why does increasing the tessellation size result in marginal changes (non-monotonic)?*\n\n**A5**: The reviewer is correct in that the tessellation size determines the degree of freedom of the velocity field. However, our ablation studies (Appendix F) demonstrate that even a small size suffices for the considered tasks. This result suggests that CPAB is highly flexible, and aligns with the conclusions in previous studies on different applications of CPAB (Martinez et al., 2022), which have shown that small sizes are sufficient in most cases.  We added this discussion to our revised paper.\n\nMartinez et al., 2022. Closed-form diffeomorphic transformations for time series alignment \n\nDubey et al., 2022. Activation functions in deep learning: A comprehensive survey and benchmark\n\nSzandala, 2021. Review and comparison of commonly used activation functions for deep neural networks", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0JKgqA8xVk", "reviewer": "Reviewer_AwEg", "summary": "Inspired by the continuous piecewise-affine based transformation, this paper argued that the activation function should not be an uniform selection for different nodes, and developed a learning activation function method, that can take the graph structure into account to activate node-specific features. The experimental results support their proposed model's effectiveness through comparing with existing activation functions in several datasets.", "questions": "1. How will you ensure the parameter \\theta you learn from the training can return you a diffeomorphism activation function? It is not clearly stated in your experimental settings, such as how you define transformation T and space \\Omega.\n2. The method used a GNN to learn the activation function, is there an activation function in the GNN_act?\n3. In the node classification experiments, there are some significant improvement in some tasks, while some of them have a minor improvement. Have you tried to figure out the reasons?", "limitations": "There is no potential negative societal impact of this work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper provides a new method to learn the activation function, where instead of activating every nodes (representations) uniformly, this method helps generate a node-specific function.\nThe experimental results prove that this method can significantly increase the model performance.", "weaknesses": "Many formula seem to be repeated, such as equations in 3.2 and 4.1.\nThe complexity is a bit high compared with an existing activation function.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful evaluation and are happy to see that the reviewer recognized the performance increase yielded by our method. We now proceed by addressing the questions and comments, and hope you find them satisfactory to consider revising your score.\n\n**Q1**: *Many formulas seem to be repeated, such as in 3.2 and 4.1. The complexity is a bit high compared with an existing activation function.*\n\n**A1**: Thank you for the feedback. Section 3.2 serves as the blueprint of the framework, where notations, GNN layers, and a general parameterized activation function are defined. Please note that each Equations (3),(4), and (5) each describe a different tensor, and are essential to define the notations that are used throughout the paper .Section 4.1 specializes in this blueprint to diffeomorphisms, in which we explain how DiGRAF transforms the node features using the CPAB framework and how this differs from traditional activation functions. We agree that some equations in Section 4.1 can be incorporated in the text instead of having a unique Equation. One example is Equation (8) that introduces the element-wise application of DiGRAF to the input node feature tensor. We followed your guidance in our revised paper.\nRegarding the complexity of the method, please note that we discuss this aspect in the limitations paragraph in our paper. Also,  we emphasize that DiGRAF exhibits linear computational complexity with respect to the input size, and can achieve sub-linear running times through parallelization, as discussed in Section 4.3. Although DiGRAF includes an additional component compared to standard activation functions, such as ReLU and Tanh, it delivers significantly better performance, as evident from our diverse experiments in Section 5. Furthermore, compared with other graph activation functions such as GReLU we note that our DiGRAF maintains faster runtimes and superior performance. We have detailed the time complexity and measured runtimes in Table 13 at Appendix H, and compared it with other methods as well. We revised the paper to better highlight this discussion. Thank you.\n\n**Q2**: *How will you ensure the parameter $\\theta$ you learn from the training can return you a diffeomorphism activation function? It is not clearly stated in your experimental settings, such as how you define transformation $T$ and space $\\Omega$.*\n\n**A2**: Thank you for the question. Please allow us to start by stating that by construction, DiGRAF yields a diffeomorphic activation function within the domain $\\Omega$, because of the utilization of the CPAB framework, which yields a diffeomorphic transformation based on  velocity field weights $\\theta$. In the case of DiGRAF, we show that the velocity field weights can be learned to be both (i) task-aware, and (ii) graph-aware. In particular, using $\\text{GNN}\\_{act}$, we learn the velocity field weights $\\theta$, which serves as the parameter for the diffeomorphism transformation built on the CPAB framework. Following the construction in the work of CPAB (Freifeld et al. 2017), the velocity field, computed using the parameter $\\theta$ and the predefined tessellation setup, is always continuous piecewise affine (CPA). To be precise, by definition, the integral of a CPA velocity field is guaranteed to be a diffeomorphic function. Therefore, the DiGRAF learned is inherently a diffeomorphic activation function. We would like to kindly note that the mathematical background for this discussion is provided in Section 3.1, and we show how to use it to learn our graph activation function DiGRAF in Section 4. In addition, we provided an overview of CPAB transformations in Appendix B.\nRegarding the definition of $T$, as defined in Equation (1) and (6), the CPAB transformation $T$ in DiGRAF at layer $l$ is $T^{(l)}(\\bar{h}^{(l)}\\_{u, c}; \\theta^{(l)}) \\triangleq f^{\\theta^{(l)}}(\\bar{h}^{(l)}_{u, c}) = \\phi^\\theta(\\bar{h}^{(l)}\\_{u, c}, t=1)$, where $\\bar{h}^{(l)}\\_{u, c}$, defined by Equation (4), represents the intermediate value of a node between the GNN layer and the activation function layer.\nRegarding the domain $\\Omega = [a, b]$, it is defined by hyperparameters $a$ and $b$ where $a < b$, as discussed in detail in Section 4. In practice, we set $a = -b$ to ensure that the activation function is symmetric and centered around 0.\nWe thank you for the questions, and we revised our text to better highlight these aspects, discussed in our paper. \nFreifeld et al. 2017. Transformations based on continuous piecewise-affine velocity fields. \n\n**Q3**: *The method used a GNN to learn the activation function, is there an activation in the $\\text{GNN}_{\\text{act}}$?*\n\n**A3**: We use the ReLU activation function within $\\text{GNN}\\_{act}$, as it is  widely used in GNNs, in order to make it non-linear. The important part is that $\\text{GNN}\\_{act}$ predicts the velocity field that defines our learned activation function. We added these details to our paper. \n\n**Q4**: *In the node classification experiments, there are some significant improvements in some tasks, while some of them have a minor improvement. Have you tried to figure out the reasons?*\n\n**A4**: We thank the reviewer for their insightful question. Following the reviewer’s suggestion, we have conducted an investigation on the differences between the datasets. Interestingly, our findings reveal that the datasets where DiGRAF showcases the largest performance improvements  (9.5% on Blog Catalog and 18.9% on Flickr) over baselines (ReLU), have more balanced label distributions, while this is not the case for the other datasets (Cora and CiteSeer, where DiGRAF nonetheless still improves by 3.6% and 1.8%). We present the distribution of these labels as Figure 1 in the additional PDF file. \n\nWe believe that further understanding the impact of the activation function and in particular of DiGRAF on imbalanced data represents an interesting future direction, which we are eager to conduct in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response

We would like to express our gratitude to all reviewers for their valuable feedback.
Overall, the reviewers appreciated the breadth and depth of our experimental analysis, defined *``extensive``* (**mZzm, SUte**) and *``comprehensive``* (**mZzm**). They found our experimental design to be *``robust``* (**SUte**), and *``adequate and convincing``* (**LFcv**). They recognized the *``significant improvements``*  (**mZzm**) over standard and graph-specific activation functions, acknowledging our method *``can significantly increase``* (**AwEg** ) the performance of GNNs.

We are also pleased to see that reviewer **LFcv** highlighted the soundness of our contributions, describing our theoretical proof as *``detailed``*.  Furthermore, reviewer **SUte** commented on the clarity of our presentation, noting that our notations and equations are *``well-defined and thoroughly explained``* with *``well-designed figures``*, and emphasized that our method addresses the *``most critical questions``* effectively (**SUte**).

Your thoughtful comments and suggestions allowed us to improve our paper, and we provided individual responses to each reviewer. We hope that you will find them satisfactory, and that you will consider revising your score. We are happy to discuss existing or additional questions and suggestions you may have. 

**New Experiments.** Several additional experiments were conducted following the reviewers’ comments, as follows:

1. A visualization of the learned activation functions for two randomly chosen graphs from the ZINC dataset, showing how $\text{GNN}_\text{act}$ adapts to different graph structures and results in different activation functions (**mZzm, SUte**); 

2. An experiment on an additional task, namely link prediction, using GCN and comparing ReLU, DiGRAF (w/o ADAP) and DiGRAF, further demonstrating DiGRAF’s  consistent improvements (**mZzm**);

3. Additional experiments using different GNN backbones beyond GCN and GINE, such as GAT and SAGE, showing DiGRAF consistently yields performance improvements, regardless of the base GNN architecture (**mZzm**), further highlighting the effectiveness of DiGRAF across multiple benchmarks and its applicability to multiple GNN backbones; 

4. Additional baselines using Sigmoid as the activation function on all tasks and datasets (**LFcv**), which we show to be consistently outperformed by our DiGRAF;

5. A comparison of the performance between DiGRAF (w/o ADAP) and DiGRAF using the same number of parameters (**SUte**). Our results in this experiment further support our findings in the paper, and in particular in Table 5, showing that the performance improvement of DiGRAF cannot be attributed to the (small) increase in the number of parameters, but rather in the way in which they are allocated to obtain graph-adaptivity.

All new experiments were discussed in their respective responses, as well as in the added rebuttal PDF. We also added all discussions and results to our revised paper.

***
References:

[1] Veličković et al., 2018. Graph Attention Networks 

[2] Hamilton et al., 2017. Inductive Representation Learning on Large Graphs

[3] Tönshoff et al., 2023. Where did the gap go? reassessing the long-range graph benchmark

[4] Martinez et al., 2022. Closed-form diffeomorphic transformations for time series alignment

[5] Scaman and Virmaux, 2018. Lipschitz regularity of deep neural networks: analysis and efficient estimation

[6] Xu and Zhang, 2023. Uniform Convergence of Deep Neural Networks with Lipschitz Continuous Activation Functions and Variable Widths

Meta-review: The contribution of this paper consists of a novel activation function in learning by GNNs.  Specifically they introduce an approach, referred to as DIGRAF exploiting Continuous Piecewise-Affine Based (CPAB) transformations augmented with an additional GNN to learn a graph-adaptive diffeomorphic activation function in an end-to-end set up. In addition to  graph-adaptivity and flexibility, DIGRAFwas show to enjoy  desirable  properties for activation functions, including  differentiability, boundness. The authors have  run experiments to demonstrate a consistent and superior performance of DIGRAF relative to traditional and graph-specific activation functions.
All reviews but one who recommended weak accept, have recommended borderline accept. 
Most of the reviewers had recommended additional experiments (ablation, more baseline and other) and the authors have for the most part  complied and have provided additional experimental results.
The most negative review stated that the real  contribution, namely  the analysis  of the connection between CPAB and GNNs needed further theoretical exploration, which, while valid, raises the question of how much can be included in a given paper.
The second comment which is also a valid one is the non-inclusion of the code for the approach, which the authors respond to, by promising to publish the code after the paper is accepted, which is clearly enforceable.  
Overall the reviews were predominantly positive.
Decision: Accept (poster)