DiGRAF: Diffeomorphic Graph-Adaptive Activation Function

Authors: krishna  sri
ipsit  mantri
aurora  wang
carola-bibiane  schönlieb
bruno  ribeiro
beatrice  bevilacqua
moshe  eliasof
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 23:48:02

Abstract: In this paper, we propose a novel activation function tailored specifically for graph data in Graph Neural Networks (GNNs). Motivated by the need for graph-adaptive and flexible activation functions, we introduce DiGRAF, leveraging Continuous Piecewise-Affine Based (CPAB) transformations, which we augment with an additional GNN to learn a graph-adaptive diffeomorphic activation function in an end-to-end manner. In addition to its graph-adaptivity and flexibility, DiGRAF also possesses properties that are widely recognized as desirable for activation functions, such as differentiability, boundness within the domain, and computational efficiency. 
We conduct an extensive set of experiments across diverse datasets and tasks, demonstrating a consistent and superior performance of DiGRAF compared to traditional and graph-specific activation functions, highlighting its effectiveness as an activation function for GNNs. Our code is available at https://github.com/ipsitmantri/DiGRAF.

Main paper: 
 introduction: 
 Graph Neural Networks (GNNs) have found application across diverse domains, including social networks, recommendation systems, bioinformatics, and chemical analysis [82, 91, 68] . Recent advancements in GNN research have predominantly focused on exploring the design space of key architectural elements, ranging from expressive GNN layers [57, 23, 88, 89, 65] , to pooling layers [87, 46, 5, 80] , and positional and structural encodings [18, 67, 20] . Despite the exploration of these architectural choices, a common trend persists where most GNNs default to employing standard activation functions, such as ReLU [26] , among a few others.Activation functions play a crucial role in neural networks, as they are necessary for modeling non-linear input-output mappings. Importantly, different activation functions exhibit distinct behaviors, and the choice of the activation function can significantly influence the performance of the neural network [60] . It is wellknown [61, 72] that, from a theoretical point of view, non-convex and highly oscillatory activation functions offer better approximation power. However, due to their strong non-convexity, they amplify optimization challenges [39] .Therefore, as a middle-ground between practice and theory, it has been suggested that a successful activation function should possess the following properties: (1) be differentiable everywhere [16, 54] , (2) have non-vanishing gradients [16] ; (3) be bounded to improve the training stability [48, 16] ; (4) be zero-centered to accelerate convergence [16] ; and (5) be efficient and not increase the complexity of the neural network [45] . In the context of graph data, the activation function should arguably also be what we define as graph-adaptive, that is, tailored LAYER to obtain updated intermediate node features H(l) , which are passed to our activation function layer, DIGRAF. First, an additional GNN network GNN ACT takes H(l) and A as input to determine the activation function parameters θ (l) . These are used to parameterize the transformation T (l) , which operates on H(l) to produce the activated node features H (l) .to the input graph and capable of capturing the unique properties of graph-structured data, such as degree differences or size changes. This adaptivity ensures that the activation function can effectively leverage the structural information present in the graph data, potentially leading to improved performance in graph tasks.Recent work in graph learning has investigated the impact of activation functions specifically designed for graphs, such as Iancu et al. [38] that proposes graph-adaptive max and median activation filters, and Zhang et al. [92] that introduces GReLU, which learns piecewise linear activation functions with a graph-adaptive mechanism. Despite the potential demonstrated by these approaches, the proposed activation functions still have predefined fixed structures (max and median functions in Iancu et al. [38] and piecewise linear in Zhang et al. [92] ), restricting the flexibility of the activation functions that can be learned. Additionally, in the case of GReLU, the learned activation functions inherit the drawback of points of non-differentiability, which are undesirable according to the properties mentioned above. As a consequence, to the best of our knowledge, none of the existing activation functions prove to be consistently beneficial across different graph datasets and tasks. Therefore, our objective is to design a flexible activation function tailored for graph data, offering consistent performance gains. This activation function should possess many, if not all, of the properties recognized as beneficial for activation functions, with an emphasis on blueprint flexibility, as well as task and input adaptivity.Our Approach: DIGRAF. In this paper, we leverage the success of learning diffeomorphisms, particularly through Continuous Piecewise-Affine Based transformations (CPAB) [24, 25] , to devise an activation function tailored for graph-structured data. Diffeomorphisms, characterized as bijective, differentiable, and invertible mappings with a differentiable inverse, inherently possess many desirable properties of activation functions, like differentiability, boundedness within the input-output domain, and stability to input perturbations. To augment our activation function with graph-adaptivity, we employ an additional GNN to derive the parameters of the learned diffeomorphism. This integration yields our node permutation equivariant activation function, dubbed DIGRAF -DIffeomorphism-based GRaph Activation Function, illustrated in Figure 1 , that dynamically adapts to different graphs, providing a flexible framework capable of learning activation functions for specific tasks and datasets in an end-to-end manner. This comprehensive set of characteristics positions DIGRAF as a promising approach for designing activation functions for GNNs.To evaluate the efficacy of DIGRAF, we conduct an extensive set of experiments on a diverse set of datasets across various tasks, including node classification, graph classification, and regression. Our evaluation compares the performance of DIGRAF with three types of baselines: traditional activation functions, activation functions with trainable parameters, and graph activation functions. Our experimental results demonstrate that DIGRAF repeatedly exhibits better downstream performance than other approaches, reflecting the theoretical understanding and rationale underlying its design and the properties it possesses. Importantly, while existing activation functions offer different behavior in different datasets, DIGRAF maintains consistent performance across diverse experimental evaluations, further highlighting its effectiveness.Main contributions. The contributions of this work are summarized as follows: (1) We introduce a learnable graph-adaptive activation function based on flexible and efficient diffeomorphisms -DIGRAF, which we show to have properties advocated in literature; (2) an analysis of such properties, reasoning about the design choices of our method; and, (3) a comprehensive experimental evaluation of DIGRAF and other activation functions.  
related work: 
 Diffeomorphisms in Neural Networks. A bijection mapping function f : M → N , given two differentiable manifolds M and N , is termed a diffeomorphism if its inverse f -1 : N → M is also differentiable. The challenge in learning diffeomorphisms arises from their computational complexity: early research is often based on complicated infinite dimensional spaces [76] , and later advancements have turned to Markov Chain Monte Carlo methods, which still suffer from large computational complexity [1, 2, 90] . To address these drawbacks, Freifeld et al. [24, 25] introduced the Continuous Piecewise-Affine Based transformation (CPAB) approach, offering a more pragmatic solution to learning diffeomorphisms by starting from a finite-dimensional space, and allowing for exact diffeomorphism computations in the case of 1D diffeomorphisms -an essential trait in our case, given that activation functions are 1D functions. CPAB has linear complexity and is parallelizable, which can lead to sub-linear complexity in practice [25] . Originally designed for alignment and regression tasks by learning diffeomorphisms, in recent years, CPAB was found to be effective in addressing numerous applications using neural networks, posing it as a suitable framework for learning transformation. For instance, Detlefsen et al. [15] learns CPAB transformations to improve the flexibility of spatial transformer layers, Martinez et al. [52] combines CPAB with neural networks for temporal alignment, Weber and Freifeld [81] introduces a novel loss function that eliminates the need for CPAB deformation regularization in time-series analysis, and Wang et al. [79] utilizes CPAB to model complex spatial transformation for image animation and motion modeling.General-Purpose Activation Functions. In the last decades, the design of activation functions has seen extensive exploration, resulting in the introduction of numerous high-performing approaches, as summarized in Dubey et al. [16] , Kunc and Kléma [45] . The focus has gradually shifted from traditional, static activation functions such as ReLU [26] , Sigmoid [45] , Tanh [35] , and ELU [11] , to learnable functions. In the landscape of learnable activation functions, the Maxout [29] unit selects the maximum output from learnable linear functions, and PReLU [33] extends ReLU by learning a negative slope. Additionally, the Swish function [66] augments the SiLU function [19] , a Sigmoid-weighted linear unit, with a learnable parameter controlling the amount of non-linearity. The recently proposed AdAct [51] learns a weighted combination of several activation functions, and DiTAC [9] learns a diffeomorphic activation function for CNNs. However, these activation functions are not input-adaptive, a desirable property in GNNs.Graph Activation Functions. Typically, GNNs are coupled with conventional activation functions [43, 78, 84] , which were not originally tailored for graph data, graph tasks, or GNN models. This implies that these activation functions do not inherently adapt to the structure of the input graph, which was found to be an important property in other GNN components, such as graph normalization [21] . Recent works have suggested various approaches to bridge this gap. Early works such as Scardapane et al. [70] propose learning activation functions based on graph kernels, and Iancu et al. [38] introduces Max and Median filters, which operate on local neighborhoods in the graph, thereby offering adaptivity to the input graphs. A notable advancement in graph-adaptive activation functions is GReLU [92] , a parametric piecewise affine activation function achieving graph adaptivity by learning parameters through a hyperfunction that takes into account the node features and the connectivity of the graph. While these approaches demonstrate the potential to enhance GNN performance compared to standard activation functions, they are constrained by their blueprint, often relying on piecewise ReLU composition, which can be performance-limiting [41] . Moreover, a fixed blueprint limits flexibility, i.e., the ability to express a variety of functions. As we show in Figure 2 , attempts to approximate traditional activation functions such as ELU and Tanh using piecewise ReLU composition with different segment counts (K = 1, 2, and 3), reveal limited approximation power. On the contrary, our DIGRAF, which leverages CPAB, yields significantly better approximations. Furthermore, we demonstrate the approximation power of activations learned with the CPAB framework in our DIGRAF in Appendix E.1. θ 1 = [1, 1, 1, 1]θ 2 =[0.5, 0.5, 0.5, 0.5]θ 3 =[-1, -1, -1, -1] Tessellation Vertices (a) -4 -2 0 2 4 x -4 -2 0 2 4 f θ (x) (b)Figure 3 : An example of CPA velocity fields v θ defined on the interval Ω = [-5, 5] with a tessellation P consisting of five subintervals. The three different parameters, θ 1 , θ 2 , and θ 3 define three distinct CPA velocity fields (Figure 3a ) resulting in separate CPAB diffeomorphisms f θ (x) (Figure 3b ). 
mathematical background and notations: 
 In this paper, we utilize the definitions from CPAB -a framework for efficiently learning flexible diffeomorphisms [24, 25] , alongside basic graph learning notations, to develop activation functions for GNNs. Consequently, this section outlines the essential details needed to understand the foundations of our DIGRAF.Let Ω = [a, b] ⊂ R be a closed interval, where a < b. We discretize Ω using a tessellation P with NP intervals, which, in practice, is oftentimes an equispaced 1D meshgrid with NP segments [25] (see Appendix C for a formal definition of tessellation). Our goal in this paper is to learn a diffeomorphism f : Ω → Ω that we will use as an activation function. Formally, a diffeomorphism is defined as follows: Definition 3.1 (Diffeomorphism on a closed interval Ω). A diffeomorphism on a closed interval Ω ⊂ R is any function f : Ω → Ω that is (1) bijective, (2) differentiable, and (3) has a differentiable inverse f -1 .To instantiate a CPAB diffeomorphism f , we define a continuous piecewise-affine (CPA) velocity field v θ parameterized by θ ∈ R N P -1 . We display examples of velocity fields v θ for various instances of θ in Figure 3a to demonstrate the distinct influence of θ on v θ . Formally, a velocity field v θ is defined as follows: Definition 3.2 (CPA velocity field v θ on Ω). Given a tessellation P with NP intervals on a closed domain Ω, any velocity field v θ : Ω → R is termed continuous and piecewise-affine if (1) v θ is continuous, and (2) v θ is an affine transformation on each interval of P.The CPA velocity field v θ defines a differentiable trajectory ϕ θ (x, t) : Ω × R → Ω for each x ∈ Ω. The trajectories are computed by integrating the velocity field v θ to time t, and are used to construct the CPAB diffeomorphism. We visualize the resulting diffeomorphism in Figure 3b with matching colors denoting corresponding pairs of v θ and f θ (x). Mathematically, Definition 3.3 (CPAB Diffeomorphism). Given a CPA velocity field v θ , the CPAB diffeomorphism f at point x, is defined as:EQUATIONsuch that ϕ θ (x, t = 1) solves the integral equation:EQUATIONIn arbitrary dimensions, computing Definition 3.3 required using an ordinary differential equation solver and can be expensive. However, for 1D diffeomorphisms, as in our DIGRAF, there are closed-form solutions to the CPAB diffeomorphism and its gradients [25] , offering an efficient framework for learning activation functions.Consider a graph G = (V, E) with N ∈ N nodes, where V = {1, . . . , N } is the set of nodes and E ⊆ V × V is the set of edges. Let A ∈ {0, 1} N ×N be the adjacency matrix of G, and X ∈ R N ×F the node feature matrix, where F is the number of input features. We denote the feature vector of node v ∈ V as xv ∈ R F , which corresponds to the v-th row of X. The input node features X are transformed into the initial node representations H (0) ∈ R N ×C , using an embedding function emb : R F → R C to X, where C is the hidden dimension, that isEQUATIONThe initial features H (0) are fed to a GNN comprised of L ∈ N layers, where each layer l ∈ {1, . . . , L} is followed by an activation function σ (l) (•; θ (l) ) : R → R, and θ (l) is a set of possibly learnable parameters of σ (l) . Specifically, the intermediate output of the l-th GNN layer is denoted as:EQUATIONwhere H(l) ∈ R N ×C . The activation function σ (l) is then applied element-wise to H(l) , yielding node features h(l) u,c = σ (l) ( h(l) u,c; θ (l) ) ∀u ∈ V , ∀c ∈ [C]. Therefore, the application of σ (l) can be equivalently written as:EQUATIONIn the following section, we will show how this abstraction is translated to our DIGRAF. 
digraf: 
 In this section, we formalize our approach, DIGRAF, illustrated in Figure 1 , which leverages diffeomorphisms to learn adaptive and flexible graph activation functions.Our approach builds on the highly flexible CPAB framework [24, 25] and extends it by incorporating Graph Neural Networks (GNNs) to enable the learning of adaptive graph activation functions. While the original CPAB framework was designed for grid deformation and alignment tasks, typically in 1D, 2D, or 3D spaces, we propose a novel application of CPAB in the context of learning activation functions, as described below. (x, q(x))CPAB: (f θ (x), q(x)))DiGRAF: (x, f θ (q(x))) (x, q(x))CPAB: (f θ (x), q(x)))DiGRAF: (x, f θ (q(x)))Figure 4 : Different transformation strategies. The input function (red), CPAB transformation (blue), and DIGRAF transformation (green), within Ω = [-5, 5] using the same θ. While CPAB stretches the input, DIGRAF stretches the output, showcasing the distinctive impact of each approach.In DIGRAF, we treat a node feature (single channel) as a one-dimensional (1D) point. Given the node features matrix H ∈ R N ×C , we apply DIGRAF per entry in H, in accordance with the typical element-wise computation of activation functions. We mention that, while CPAB was originally designed to learn grid deformations, it can be utilized as an activation function blueprint by considering a conceptual shift that we demonstrate in Figure 4 . Given an input function (shown in red in the figure), CPAB deforms grid coordinates, i.e., it transforms it along the horizontal axis, as shown in the blue curve. In contrast, DIGRAF transforms the original data points along the vertical axis, resulting in the green curve. This conceptual shift can be seen visually from the arrows showing the different dimensions of transformations. We therefore refer to the vertical transformation of the data as their activations. Formally, we define the transformation function T (l) as the element-wise application of the diffeomorphism f θ from Equation (1):EQUATIONwhere θ (l) denotes learnable parameters of the transformation function T (l) , that parameterize the underlying CPA velocity field as discussed in Section 3. In Section 4.2 , we discuss the learning of θ (l) in DIGRAF.The transformation T (l) : Ω → Ω described in Equation ( 6) is based on CPAB and therefore takes as input values within a domain Ω = [a, b], and outputs a value within that domain, where a < b are hyperparameters.In practice, we take a = -b, such that the activation function can be symmetric and centered around 0, a property known to be desirable for activation functions [16] . For any entry in the intermediate node features H(l) (Equation ( 4)) that is outside the domain Ω, we use the identity function. Therefore, a DIGRAF activation function reads:EQUATIONIn practice, DIGRAF is applied element-wise in parallel over all entries, and we use the following notation, which yields the output features post the activation of the l-th GNN layer:EQUATIONDIGRAF, defined in Equation ( 7), introduces graph-adaptivity into the transformation function T (l) by employing an additional GNN, denoted as GNNACT, that returns the diffeomorphism parameters θ (l) :EQUATIONwhere POOL is a graph-wise pooling operation, such as max or mean pooling. The resulting vector θ (l) ∈ R N P -1 , which is dependent on the tessellation size NP , is then used to compute the output of the l-th layer, H (l) , as described in Equation (8) . We note that Equation ( 9) yields a different θ (l) for every input graph and features pair ( H(l) , A), which implies the graph-adaptivity of DIGRAF. Furthermore, since GNNACT is trained with the other network parameters in an end-to-end fashion, DIGRAF is also adaptive to the task of interest. In Appendix B, we provide and discuss the implementation details of GNNACT and POOL.Variants of DIGRAF. Equation ( 9) describes an approach to introduce graph-adaptivity to θ (l) usingGNNACT. An alternative approach is to directly optimize the parameters θ (l) ∈ R N P -1 , without using an additional GNN. Note that in this case, input and graph-adaptivity are sacrificed in favor of a computationally lighter solution. We denote this variant of our method by DIGRAF (W/O ADAP.). Considering this variant is important because it allows us to: (i) offer a middle-ground solution in terms of computational effort, and (ii) it allows us to directly quantify the contribution of graph-adaptivity in DIGRAF. In Section 5, we compare the performance of the methods.Velocity Field Regularization. To ensure the smoothness of the velocity field, which will encourage training stability [81] , we incorporate a regularization term in the learning procedure of θ (l) . Namely, we follow the Gaussian smoothness prior on the CPA velocity field from Freifeld et al. [24] , which was shown to be effective in maintaining smooth transformations. The regularization term is defined as follows:EQUATIONwhere ΣCPA represents the covariance of a zero-mean Gaussian smoothness prior defined as in Freifeld et al. [24] . We further maintain the boundedness of θ (l) by employing a hyperbolic tangent function (Tanh). In this way, θ (l) remains in [-1, 1] when applied in T (l) in Equation (7) , ensuring that the velocity field parameters remain bounded, encouraging the overall training stability of the model.In this section, we focus on understanding the theoretical properties of DIGRAF, highlighting the compelling attributes that establish it as a performant activation function for GNNs.DIGRAF yields differentiable activations. By construction, DIGRAF learns a diffeomorphism, which is differentiable by definition. Being differentiable everywhere is considered beneficial as it allows for smooth weight updates during backpropagation, preventing the zigzagging effect in the optimization process [77] .DIGRAF is bounded within the input-output domain Ω. We point out in Remark D.3 that the diffeomorphismT (l) (•; θ (l)) is a Ω → Ω transformation. Any diffeomorphism is continuous, and by the extreme value theorem, T (l) (•; θ (l) ) is bounded in Ω. This prevents the activation values from becoming excessively large, a property linked to faster convergence [16] .DIGRAF can learn to be zero-centered. Benefiting from its flexibility, DIGRAF has the capacity to learn activation functions that are inherently zero-centered. As an input-adaptive activation function governed by a parameters vector θ (l) , DIGRAF can be adjusted through θ (l) to maintain a zero-centered nature. This property is associated with accelerated convergence in neural network training [16] .DIGRAF is efficient. DIGRAF exhibits linear computational complexity, and can further achieve sub-linear running times via parallelization in practice [25] . Moreover, with the existence of a closed-form solution for f θ (l) and its gradient in the 1D case [24] , the computations of CPAB can be done efficiently. Additionally, the measured runtimes, detailed in Appendix H, underscore the complexity comparability of DIGRAF with other graph activation functions.In addition to the above properties, which follow from our design choice of learning diffeomorphisms through the CPAB framework, we briefly present the following properties, which are formalized and proven in Appendix D.DIGRAF is permutation equivariant. We demonstrate in Proposition D.4 that DIGRAF exhibits permutation equivariance to node numbering, ensuring that its behavior remains consistent regardless of the ordering of the graph nodes, which is a key desired property in designing GNN components [8] .DIGRAF is Lipschitz continuous. We show in Proposition D.2 that DIGRAF is Lipschitz continuous and derive its Lipschitz constant. Since it is also bounded, we can combine the two results, which leads us to the following proposition: Proposition 4.1 (The boundedness of T (•; θ (l) ) in DIGRAF). Given a bounded domain Ω = [a, b] ⊂ R where a < b, and any two arbitrary points x, y ∈ Ω, the maximal difference of a diffeomorphism T (•; θ (l) ) with parameter θ (l) in DIGRAF is bounded as follows:EQUATIONwhereC v θ (l)is the Lipschitz constant of the CPA velocity field v θ (l) .DIGRAF extends commonly used activation functions. CPAB [24, 25] , which is used as a framework to learn the diffeomorphism in DIGRAF, is capable of learning and representing a wide range of diffeomorphic functions. When used as an activation function, the transformation T (l) (•; θ (l) ) in DIGRAF adapts to the specific graph and task by learning different θ (l) parameters, rather than having a fixed diffeomorphism. Examples of popular and commonly used diffeomorphisms utilized as activations include Sigmoid, Tanh, Softplus, and ELU, as we show in Appendix D. Extending this approach is our DIGRAF that learns the diffeomorphism during training rather than selecting a pre-defined function.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: See Section 5, Appendix E, Appendix B and Appendix G.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
experiments: 
 In this section, we conduct an extensive set of experiments to demonstrate the effectiveness of DIGRAF as a graph activation function. Our experiments seek to address the following questions: Baselines. We compare DIGRAF with three categories of relevant and competitive baselines: (1) Standard Activation Functions, namely Identity, Sigmoid [69] , ReLU [26] , LeakyReLU [50] , Tanh [35] , GeLU [34] ,and ELU [12] to estimate the benefit of learning activation functions parameters; (2) Learnable Activation Functions, specifically PReLU [33] , Maxout [29] and Swish [66] , to assess the value of graph-adaptivity; and(3) Graph Activation Functions, such as Max [38] , Median [38] and GReLU [92] , to evaluate the effectiveness of DIGRAF's design in capturing graph structure and the blueprint flexibility of DIGRAF as discussed in Section 4.All baselines are integrated into GCN [43] for node tasks and GIN [84] (GINE [37] where edge features are available) for graph tasks, to ensure fair and meaningful comparisons, isolating the impact of other design choices. We provide additional details on the experimental settings and datasets in Appendix G, as well as additional experiments, including ablation studies, in Appendix E.Our results are summarized in Table 1 , where we consider the BLOGCATALOG [86] , FLICKR [86] , CITE-SEER [73] , CORA [53] , and PUBMED [58] datasets. As can be seen from the Table, DIGRAF consistently outperforms all standard activation functions, as well as all the learnable activation functions. Additionally, DIGRAF outperforms other graph-adaptive activation functions. We attribute this positive performance gap to the ability of DIGRAF to learn complex non-linearities due to its diffeomorphism-based blueprint, compared to piecewise linear or pre-defined functions as in other methods. Finally, we compare the performance of DIGRAF and DIGRAF (W/O ADAP.). We remark that in this experiment, we are operating in a transductive setting, as the data consists of a single graph, implying that both DIGRAF and DIGRAF (W/O ADAP.) are adaptive in this case. Still, we see that DIGRAF slightly outperforms the DIGRAF (W/O ADAP.) and we attribute this performance gain to the GNN layers within DIGRAF that are (i) explicitly graph-aware, and (ii) can facilitate the learning of better diffeomorphism parameters θ (l) (Equation ( 9)) due to the added complexity.Table 2 : Comparison on ZINC-12K under the 500K parameter budget. The top three methods are First, Second, Third.STANDARD ACTIVATIONS GIN + Identity 0.2460±0.0214 GIN + Sigmoid [69] 0.3839±0.0058 GIN + ReLU [84] 0.1630±0.0040 GIN + LeakyReLU [50] 0.1718±0.0042 GIN + Tanh [35] 0.1797±0.0064 GIN + GeLU [34] 0.1896±0.0023 GIN + ELU [12] 0.1741±0.0089GIN + PReLU [33] 0.1798 ±0.0067 GIN + Maxout [29] 0.1587±0.0057 GIN + Swish [66] 0.1636±0.0039GIN + Max [38] 0.1661±0.0035 GIN + Median [38] 0.1715±0.0050 GIN + GReLU [92] 0.3003±0.0086GIN + DIGRAF (W/O ADAP.) 0.1382±0.0080 GIN + DIGRAF 0.1302±0.0090 ZINC-12K.In Table 2 we present results on the ZINC-12K [75, 31, 18] dataset for the regression of constrained solubility of molecules. We note that DIGRAF achieves an MAE of 0.1302, surpassing the best-performing activation on this dataset, Maxout, by 0.0285, which translates to a relative improvement of ∼ 18%.OGB. We evaluate DIGRAF on 4 datasets from the OGB benchmark [36] , namely, MOLESOL, MOLTOX21, MOLBACE, and MOL-HIV. The results are reported in Table 3 , where it is noted that DIGRAF achieves significant improvements compared to standard, learnable, and graph-adaptive activation functions. For instance, DIGRAF obtains a ROC-AUC score of 80.28% on MOLHIV, an absolute improvement of 4.7% over the best performing activation function (ReLU).TUDatasets.. In addition to the aforementioned datasets, we evaluate DIGRAF on the popular TUDatasets [56] . We present results on MUTAG, PTC, PROTEINS, NCI1 and NCI109 in Table 5 in Appendix E. The results show that DIGRAF is always within the top-three performing activations across all datasets.As an example, on PROTEINS dataset, we see an absolute improvement of 1.1% over the best-performing activation functions (Maxout and GReLU).Besides improved downstream performance, another important aspect of activation functions is their contribution to training convergence [16] . We therefore present the training curves of DIGRAF as well as the rest of the considered baselines to gain insights into their training convergence. Results for representative datasets are presented in Figure 5 , where DIGRAF achieves similar or better training convergence than other methods, while also demonstrating better generalization abilities due to its better performance.Our extensive experiments span across 15 different datasets and benchmarks, consisting of both node-and graph-level tasks. Our key takeaways are as follows: (A1) Overall Performance of DIGRAF: The performance offered by DIGRAF is consistent and on par with or better than other activation functions, across all datasets. These results establish DIGRAF as a highly effective approach for learning graph activation functions.(A2) Benefit of Graph-Adaptivity: DIGRAF outperforms the learnable (although not graph-adaptive) activation functions such as PReLU, Maxout, and Swish, as well as our non-graph adaptive baseline DIGRAF (W/O ADAP.), on all considered datasets. This observation highlights the crucial role of graph-adaptivity in activation functions for GNNs.(A3) The Benefit of Blueprint Flexibility: DIGRAF consistently outperforms other graph-adaptive activation functions like Max, Median, and GReLU. We tie this positive performance gap to the ability of DIGRAF to model complex non-linearities due to its diffeomorphism-based blueprint, compared to piecewise linear or pre-defined functions as in other methods.(A4) Convergence of DIGRAF: As shown in Section 5.3, in addition to overall better downstream performance, DIGRAF allows to achieve better training convergence.In summary, compared with 12 well-known activation functions used in GNNs, and across multiple datasets and benchmarks, DIGRAF demonstrates a superior, learnable, flexible, and versatile graph-adaptive activation function, highlighting it as a strong approach for designing and learning graph activation functions.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]Justification: We provide extensive details on the implementation and evaluation of our method, and we released our code.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusions: 
 In this work, we introduced DIGRAF, a novel activation function designed for graph-structured data. Our approach leverages Continuous Piecewise-Affine Based (CPAB) transformations to integrate a graph-adaptive mechanism, allowing DIGRAF to adapt to the unique structural features of input graphs. We show that DIGRAF exhibits several desirable properties for an activation function, including differentiability, boundedness within a defined interval, and computational efficiency. Furthermore, we demonstrated that DIGRAF maintains stability under input perturbations and is permutation equivariant, therefore suitable for graph-based applications.Our extensive experiments on diverse datasets and tasks demonstrate that DIGRAF consistently outperforms traditional, learnable, and existing graph-specific activation functions.Limitations and Broader Impact. While DIGRAF demonstrates consistent superior performance compared to existing activation functions, there remain areas for potential improvement. For instance, the current formulation is limited to learning activation functions that belong to the class of diffeomorphisms, which, despite encompassing a wide range of functions, might not be optimal. By improving the performance on real-world tasks like molecule property prediction, and offering faster training convergence, we envision a positive societal impact by DIGRAF in drug discovery and in achieving a lower carbon footprint.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: See Appendix G and Section 5.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: The standard deviations for all the metrics have been presented. See Table 1 , Table 2 , Table 3 , Table 5 , Table 7 , Table 8 , Table 9 , and Table 16 . Also see Section 5, Appendix E. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We discuss the computational resources in Appendix G and Appendix H.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cited each datasets we used in Section 5 and Appendix G. The license of the packages we used is in Appendix G.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: We don't release any new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? 
 
 Appendix: Graph Neural Networks. Graph Neural Networks [71] (GNNs) have emerged as a transformative approach in machine learning, notably following the popularity of the message-passing scheme [28] . GNNs enable effective learning from graph-structured data, and can be applied to different tasks, ranging from social network analysis [43] to bioinformatics [40] . In recent years, various GNN architectures were proposed, aiming to address various aspects, from alleviating oversmoothing [10] , concerning attention mechanisms in the message passing scheme [78, 7, 42] , or focusing on the expressive power of the architectures [57, 23, 88, 89, 65, 4] , given that message-passing based architectures are known to be bounded by the WL graph isomorphism test [84, 55] .Despite advancements, the poor performance of deep GNNs has led to a preference for shallow architectures GCNs [47] . To enhance performance, techniques such as pooling functions have been proposed, introducing generalization by reducing feature map sizes [93] . Methods such as HGP-SL [93] , GraphUNet [27] , and LaPool [59] introduce pooling layers specifically designed for GNNs. Beyond node feature, the importance of graph structure and positional features is increasingly recognized, with advancements such as GraphGPS [67] and SAN [44] integrating positional and structural encodings through attention-based mechanisms.Evaluation of Rectified Activation Functions. Rectified activation functions, represented by the Rectified Linear Unit (ReLU), have been widely applied and studied in various neural network architectures due to their simplicity and effectiveness [60, 3, 45] . The prevalent assumption that ReLU's performance is predominantly due to its sparsity is critically examined by Xu et al. [83] , suggesting introducing a non-zero slope in the negative part can significantly enhance network performance. Extending this, Price et al. [64] investigates sparsity-inducing activation functions, such as the shifted ReLU, in network initialization and early stages of training. These functions can mitigate overfitting and boost model generalization capabilities. Conversely, it was shown that in overparameterized networks, smoother activation functions, like Tanh and Swish, can enhance the convergence rate, in contrast to the non-smooth characteristics of ReLU [62] . However, the fixed nature of ReLU and many of its variants restricts their ability to adapt the input, resulting in limited power to capture dynamics in learning.Advancements in Learnable Activation Functions. Recent research has increasingly focused on adaptive and learnable activation functions, which are optimized alongside the learning process of the network. The AdAct framework [51] introduces learnability by combining multiple activation functions into a single module with learnable weighting coefficients. However, these coefficients are fixed after training, limiting the framework's adaptability to varying inputs. A concurrent work by Liu et al. [49] introduces Kolmogorov-Arnold Networks (KAN), a novel architecture that diverges from traditional Multi-Layer Perceptron (MLP) configurations, which applies activation functions to network edges instead of nodes. Unlike our current work, which focuses only on the design of activation functions for GNNs, their research extends beyond this scope and considers a fundamental architecture design. Finally, the recently proposed TAFS [85] learns a task-adaptive (but not graph-adaptive) activation function for GNNs through a bi-level optimization. To create a unified feature matrix for S that encompasses all graphs in the batch, we standardize the dimension by padding each feature matrix Xi ∈ R N i ×C , i ∈ [B] for graph Gi ∈ S from Ni to Nmax with zeros. The combined feature matrix XS is constructed by concatenating the transposed feature matricesMultipleX ⊤ i ∀i ∈ [B], resulting in a matrix that lies in the domain R (B•C)×Nmax . This matrix is permutation invariant; while relabeling nodes changes the row indices, it does not affect the overall transformation process. Therefore, DIGRAF can handle multiple graphs in a batch. In practice, to avoid the overhead of padding, we use the batching support from Pytorch-Geometric [22] .Implementation Details of GNN ACT . In Section 4.2, we examined two distinct approaches to learn the diffeomorphism parameters θ (l) , either directly or through GNNACT. As shown in Appendix C, θ (l) determines the velocity field v θ (l) . Predicting a graph-dependent θ (l) adds graph-adaptivity to the activation function T (l) . In DIGRAF we achieve this by employing another GNN GNNACT, described below.The backbone of GNNACT utilizes the same structure as the primary network layers GNN (l) LAYER , that is, GCN [43] or GIN [84] . It is important to note, that while GNNACT has a similar structure to the primary network GNN with ReLU activation function, it has its own set of learnable weights, and it is shared among the layers, unlike the primary GNN layers GNN (l) LAYER . The hidden dimensions and the number of layers of GNNACT are hyperparameters. The weight parameters of GNNACT are trained concurrently with the main network weights. As described in Equation ( 9), after the computation of GNNACT, a pooling layer denoted by POOL is placed to aggregate node features. This aggregation squashes the node dimension such that the output is not dependent on the specific order of nodes, and it yields the vector of parameters θ (l) .Rescaling H(l) . Following the implementation of Freifeld et al. [24] , the default 1D domain for CPAB is set as [0, 1]. To enhance the flexibility of T (l) and ensure its adaptability across various input datasets, DIGRAF extends the domain to Ω = [a, b] ⊂ R with a < b as shown in Section 3.1. To match the two domains, we rescale the intermediate feature matrix H(l) from Ω to the unit interval [0, 1] before passing it to T (l) . Let r = b-a 2 , then rescaling is performed using the function f (x) = (x + r)/(2r). Data points outside this range will retain their original value, effectively acting as an identity function outside the domain Ω.Training Loss Function. As described in Equation ( 10), we employ a regularization term for the velocity field to maintain the smoothness of the activation function. To control the strength of regularization, we introduce a hyperparameter λ. We denote LTASK as the loss function of the downstream task (i.e. cross-entropy loss in case of classification and mean absolute error in case of regression tasks), and the overall training loss of DIGRAF, denoted as LTOTAL is given asEQUATIONC Overview of CPA Velocity Fields and CPAB TransformationsIn this Section, we drop the layer notations l for simplicity. In Section 3.1, we introduce the concept of a diffeomorphism on a closed interval in Definition 3.1, which can be learned through the integration of a Continuous Piecewise Affine (CPA) velocity field. As detailed in Definition 3.2, the velocity field v θ is governed by the parameter θ and the tessellation P. We now discuss how the velocity fields are computed following the methodologies presented by Freifeld et al. [24, 25] and highlight the relations between v θ , θ and P. We start by formally defining the tessellation on Ω: Definition C.1 (Tessellation of a closed interval [24] ). A tessellationP of size NP subintervals of a closed interval Ω = [a, b] in R is a partitioning {[xi, xi+1]} N P -1 i=0that satisfies the following properties: EQUATIONThe vector of parameters θ is linked to the subintervals in P, whose dimension is determined by the number of intervals NP . Similar to Freifeld et al. [24] , we impose boundary constraints that mandate the velocity at the boundary of the tessellation to be zero, i.e., v θ (0) = v θ (1) = 0. This boundary condition allows us to compose the diffeomorphism in the domain Ω with an identity function for any values outside the domain. Under this constraint, the degrees of freedom (number of parameters) for θ is NP -1.The velocity field is then defined as follows: Definition C.2 (Relation between θ and v θ , taken from Freifeld et al. [25] ). Given a tessellation P with NP intervals on a closed domain Ω = [a, b], as defined in Definition C.1. Given a parameter θ ∈ R N P -1 and an arbitrary point x within the domain, a continuous piecewise-affine velocity field v θ can present as follows:EQUATIONwhere{bj} N P -2 j=0is an orthonormal basis of the space of velocity fields V, such that v θ ∈ V, and x = x 1 .The orthonormal basis {bj} N P -2 j=0 for the velocity field can be obtained through Singular Value Decomposition of L. Note that L is a matrix constraining the coefficients of each continuous piecewise-affine velocity function by ensuring that the velocity value at the shared endpoints is the same [52] . Let vec(A) be a column vector containing the coefficients for each interval, for instance, [a0, b0, a1, b1] T for consecutive intervals interval 0 and interval 1. The shared endpoint is x1. To achieve the constrain, we have the equation a0 * x1 + b0 + a1 * (-x1) + b1 * (-1) = 0. In this example, the constrain matrix L is L = [x1, 1, -x1, -1].× × ✓ ✓ ✓ × Tanh [35] ✓ ✓ ✓ ✓ ✓ × PReLU [33] × × ✓ ✓ ✓ × Swish [66] × ✓ ✓ ✓ ✓ × Max [38] × × ✓ ✓ ✓ ✓ Median [38] × × ✓ * ✓ - ✓ GReLU [92] × × ✓ ✓ - ✓ DIGRAF ✓ ✓ ✓ ✓ ✓ ✓By generalizing the previous example, the constraint can be expressed as L * vec(A) = -→ 0 . As the endpoints are decided by the tessellation setup, we can build the constrain matrix L without knowing vec(A). And thus, orthonormal basis {bj} N P -2 j=0 can be computed by giving tessellation setup.Proposition C.3 (DIGRAF has a closed form solution). Equation 2 can be expressed as an equivalent ODE. By allowing x to vary and fixing t, the solution to this ODE can be written as a composition of a finite number of solutions ψ:ϕ θ (x, t) = (ψ tm θ,cm • ψ t m-1 θ,c m-1 • • • • • ψ t 2 θ,c 2 • ψ t 1 θ,c 1 )(x)Here m represents the number of cells visited. Given x, θ, time t, and the smallest cell index containing x, c, we can compute each ψ t i θ,c i (x), i ∈ {1, . . . , m} from ψ t 1 θ,c 1 (x) to ψ tm θ,cm (x). In other words, DIGRAF has a closed-form solution.Proof. The proof follows the steps in Martinez et al. [52] . Equation ( 2) can be expressed as the equivalent ODE: dϕ θ (x,t) dt = v θ (ϕ θ (x, t)). By allowing x to vary and fixing t, the solution to this ODE can be written as a composition of a finite number of solutions ψ:ϕ θ (x, t) = (ψ tm θ,cm • ψ t m-1 θ,c m-1 • • • • • ψ t 2 θ,c 2 • ψ t 1 θ,c 1 )(x)Given x, θ, time t, and a function γ that returns the smallest cell index containing x, namely c = γ(x), then we can compute each ψ t i θ,c i (x), i ∈ {1, . . . , m} and use them in the closed form solution for the ODE. The cell boundary xc is determined based on the velocity value v(x) at point x. If v(x) ≥ 0, xc is the largest point in the interval; otherwise, it is the smallest point. In this setup, a cell is a 1D interval with two endpoints. At the hitting time t hit , ψ θ c (x, t hit ) isψ θ c (x, t hit ) = xc, where t θ hit = 1 a θ c log a θ c xc+b θ c a θ c x+b θ c. The CPAB velocity field is continuous piecewise-affine, and for each interval with index c, it has coefficients a θ c (slope) and b θ c (bias). These can be computed given θ. If t θ hit > t, then ϕ θ (x, t) = ψc(x, t). Otherwise, we repeat the process with updated values t = t -t θ hit , x = xc, and c adjusted based on the sign of v(x).This iterative process continues until convergence, with an upper bound for m being max(c1, NP -c1 + 1), where c1 refers to the first visited cell index, and NP is the number of closed intervals in the space Ω. With the above steps, we can precisely compute eachψ t i θ,c i (x), i ∈ {1, . . . , m} from ψ t 1 θ,c 1 (x) to ψ tm θ,cm (x) following the equation ϕ θ (x, t) = (ψ tm θ,cm • • • • • ψ t 1 θ,c 1 )(x). This allows us to determine the exact solution for ϕ θ (x, t).We present a summary of the properties offered by our DIGRAF that are absent in general-purpose activation functions or existing graph activations in Table 4 .Similar to Appendix C, for simplicity, in this Section, we drop the layer notations l.In this section, we present the propositions and proofs for the properties outlined in Section 4.3. We begin by remarking that as shown in Section 4.3, DIGRAF is bounded within the domain Ω = [a, b], where a < b by construction. We then present Proposition D.1 that outlines the Lipschitz constant of the velocity field v θ , followed by Proposition D.2, showing that DIGRAF is also Lipschitz continuous, and provide an upper bound on its Lipschitz constant.Proposition D.1 (The Lipschitz Constant of v θ ). Given two arbitrary points x, y ∈ R, and velocity field parameters θ ∈ R N P -1 that define the continuous piecewise-affine velocity field v θ , there exists a Lipschitz constantEQUATIONwhere | • | and ∥ • ∥2 denote the absolute value of a scalar and the ℓ2 norm of a vector, respectively.Proof. First, we note that it was shown in Freifeld et al. [24, 25] that v θ is Lipschitz continuous, and now we provide a derivation of that Lipschitz constant. Following Definition C.2, the velocity field v θ is defined as v θ (x) = N P -2 j=0 θjbj x, where {bj} N P -2 j=0 is an orthonormal basis of the velocity space. By the definition of v θ (x) and v θ (y), we have the following:EQUATION=EQUATIONEQUATIONEQUATION= ∥( xỹ)∥2EQUATIONEQUATIONwhere the transition between Equation ( 16) and Equation ( 17) follows from the triangle inequality, and the transition between Equation (17) and Equation (18) follows from bj being an orthonormal vector.From the derivation above, and the fact that we know from Freifeld et al. [24, 25] that the velocity field is Lipschitz continuous, we conclude that the Lipschitz constantC v θ of v θ reads C v θ = N P -2 j=0 |θj|.Given the Lipschitz constant C v θ for v θ , we proceed to demonstrate that the transformation T (•; θ) in DIGRAF is Lipschitz continuous, as well as bounding its Lipschitz constant. Proposition D.2 (The Lipschitz Constant of DIGRAF). The diffeomorphic function T (•; θ) in DIGRAF is defined in Equation (6) for a given set of weights θ, which in turn define the velocity field v θ . Let x, y ∈ R be two arbitrary points, then the following inequality is satisfied:EQUATIONwhereC v θ is the Lipschitz constant of v θ .Proof. We begin by substituting T (•; θ) with Equation ( 1) and Equation (6) . Utilizing Proposition D.1, we then establish an upper bound for |T (x; θ) -T (y; θ)| as follows:EQUATIONEQUATIONwhere C v θ is the Lipschitz constant of v θ (Proposition D.1) and the last transition follows from Grönwall's inequality [30] . Consequently, the Lipschitz constant of DIGRAF is bounded from above by exp(C v θ ).Now that we established that T (•; θ) is Lipschitz continuous and presented an upper bound, we investigate what is the maximal difference in the output of T (•; θ) with respect to two arbitrary inputs x, y ∈ Ω, and whether it can be bounded. To address this, we present the following remark:Remark D.3. Given a bounded domain Ω = [a, b], a < b, by construction, the diffeomorphism T (•; θ) with parameter θ in DIGRAF, as in Equation ( 7), is a Ω → Ω transformation [24, 25] . Therefore, by the max value theorem, the maximal output discrepancy for arbitrary x, y ∈ Ω is |b -a|, i.e., |T (x; θ) -T (y; θ)| ≤ |b -a|.Combining the Proposition D.1, Proposition D.2 and Remark D.3, we formalize and prove the following proposition:Proposition 4.1 (The boundedness of T (•; θ (l) ) in DIGRAF). Given a bounded domain Ω = [a, b] ⊂ R where a < b, and any two arbitrary points x, y ∈ Ω, the maximal difference of a diffeomorphism T (•; θ (l) ) with parameter θ (l) in DIGRAF is bounded as follows:EQUATIONwhere C v θ (l) is the Lipschitz constant of the CPA velocity field v θ (l) .Proof. In Proposition D.2 we presented an upper bound on the Lipschitz constant of T (•; θ), and in D.3 we also presented an upper bound on the maximal difference between the application of T (•; θ) on two inputs x, y.Combining the two bounds, we get the following inequality:EQUATION)The result in Proposition 4.1 gives us a tighter upper bound on the boundedness of the transformation T (•; θ) in our DIGRAF that is related both to the hyperparameters a, b, as well as the learned velocity field parameters θ.Next, we discuss another property outlined in Section 4.3, demonstrating that DIGRAF is permutation equivariant -a desirable property when designing a GNN component [8] . Proposition D.4 ( DIGRAF is permutation equivariant.). Consider a graph encoded by the adjacency matrix A ∈ R N ×N , where N is the number of nodes. Let H(l) ∈ R N ×C be the intermediate node features at layer l, before the element-wise application of our DIGRAF. Let P be an N × N permutation matrix. Then,EQUATIONEQUATIONwhere θ (l)P and θ (l) are obtained by feeding P H(l) and H(l) , respectively, to Equation (9).Proof. We break down the proof into two parts. First, we show that GNNACT outputs the same θ under permutations, that is we show θ (l) P = θ (l) . Second, we prove that the activation function T (l) is permutation equivariant, ensuring the overall method maintains this property.To begin with, recall that Equation ( 9) is composed by GNNACT, which is permutation equivariant, and by a pooling layer, which is permutation invariant. Therefore their composition is permutation invariant, that is θ (l) P = θ (l) . Prior to the activation function layer T (l) , H(l) undergoes rescaling as described in Appendix B, which is permutation equivariant as it operates element-wise. Finally, since activation function T (l) acts element-wise, and given that θ remains unchanged, the related CPA velocity fields are identical, resulting in the same transformed output for each entry, despite the entries being permuted in P H(l) . Therefore, DIGRAF is permutation equivariant.In this section, we provide several examples of popular and well-known diffeomorphic functions, contributing to our motivation to utilize diffeomorphisms as a blueprint for learning graph activation functions. We remark that, differently from standard activation functions, our DIGRAF does not need to follow a predefined, fixed template, but can instead learn a diffeomorphism best suited for the task and input, as T (l) within CPAB can represent a wide range of diffeomorphisms [24, 25] .We recall that, as outlined in Section 3.1, a function is classified as a diffeomorphism if it is (1) bijective, (2) differentiable, and (3) has a differentiable inverse.Sigmoid. We denote the Sigmoid activation function as σ : R → (0, 1), defined byσ(x) = 1 1 + e -x .To prove that σ is a diffeomorphism, we first establish its bijectivity. Injectivity follows from observing that for any distinct points x1 and x2 in R, σ(x1) =1 1+e -x 1 can only equal σ(x2) = 1 1+e -x 2 if and only if x1 = x2.For surjectivity, we represent x as a function of y, such that y = 1 1+e -x =⇒ x = -ln 1-y y , ensuring that for every y ∈ (0, 1) there is an element x ∈ R such that σ(x) = y.To demonstrate differentiability, we examine the derivative of σ. The derivatived dx σ(x) = σ(x)(1 -σ(x)),which is continuous. Additionally, the inverse functionσ -1 (y) = -ln 1 -y yis also bijective and differentiable. Thus, with all these requirements satisfied, σ is indeed a diffeomorphism.Tanh. The hyperbolic tangent function tanh(x) = e x -e -x e x + e -x is a diffeomorphism from R to (-1, 1). To establish this, we demonstrate that tanh is bijective and differentiable, with a differentiable inverse function.Firstly, tanh is injective because if tanh(x1) = tanh(x2), then x1 = x2. It is also surjective because for any y ∈ (-1, 1), there exists x = 1 2 ln 1+y1-y such that tanh(x) = y.The derivatived dx tanh(x) = 1 -tanh 2 (x)is continuous and positive. Additionally, the inverse functiontanh -1 (y) = 1 2 ln 1 + y 1 -yis continuously differentiable. Therefore, tanh qualifies as a diffeomorphism.Softplus. To establish the Softplus functionsoftplus(x) = ln(1 + e x )as a diffeomorphism from R to (0, ∞), we first demonstrate its injectivity and surjectivity.Assuming softplus(x1) = softplus(x2), we obtain e x 1 = e x 2 , implying x1 = x2, hence establishing injectivity.For any y ∈ (0, ∞), we find an x ∈ R such that y = ln(1 + e x ), ensuring surjectivity.The derivative of the Softplus function, d dx softplus(x) = e x 1 + e x = σ(x), where σ(x) is the Sigmoid function, known to be continuous and differentiable. Therefore, softplus(x) is continuously differentiable.Considering the inverse of the Softplus function, softplus -1 (y) = ln(e y -1), its derivative is d dy softplus -1 (y) = e y e y -1 , which is continuous for all y > 0, indicating that softplus -1 (y) is continuously differentiable for all y > 0.Therefore, we conclude that the softplus function qualifies as a diffeomorphism.ELU. The ELU activation function [11] is defined as below:ELU(x) = x if x > 0 α(e x -1) if x ≤ 0where α ∈ R is a constant that scales the negative part of the function. To demonstrate that ELU is bijective, we analyze its injectivity and surjectivity. For x > 0, ELU acts as the identity function, which is inherently injective. For x ≤ 0, α(e x 1 -1) = α(e x 2 -1), implies x1 = x2. The inverse function for ELU is given by:ELU -1 (y) = y if y > 0 ln( y α + 1) if y ≤ 0This inverse maps every value in the codomain back to a unique value in the domain, proving that ELU is surjective.Next, we examine the continuity of ELU. At x = 0, ELU(x = 0) = α(e 0 -1) = 0. Next, we check the limits for both sides of 0. For x > 0, lim x→0 + ELU(x) = lim x→0 + x = 0, while for x ≤ 0, we have lim x→0 -ELU(x) = lim x→0 -α(e x -1) = 0. Since both limits are equal, the ELU function is continuous at x = 0. For the derivative of ELU, i.e.,d dx ELU(x) = 1 if x > 0 αe x if x ≤ 0 at x = 0, we have d dx ELU(x) = αe 0 = α.By setting α = 1, the derivative at x = 0 matches the derivative for x > 0, making the derivative continuous.The derivative for the inverse function isd dy ELU -1 (y) = 1 if y > 0 1 y+α if y ≤ 0which is also continuously differentiable. Hence, ELU is a diffeomorphism. The combination of learned linear layers together with non-linear functions such as ReLU and Tanh are wellknown to yield good function approximations [13, 14] . Therefore, when designing an activation function blueprint, i.e., the template by which the activation function is learned, it is important to consider its approximation power. In Section 1 and in particular in Figure 2 , we demonstrate the ability of the CPAB framework to approximate known activation functions. We now show additional evidence for the flexibility and power of CPAB as a framework for learning activation functions, leading to our DIGRAF. To this end, we consider the ability of a multilayer perceptron (MLP) with various activation functions (ReLU, Tanh, and DIGRAF) to approximate the well-known 'peaks' function that mathematically reads:EQUATIONThe peaks function in Equation ( 27) is often times used to measure the ability of methods to approximate functions [32] , where the input is point pairs (x, y) ∈ R 2 , and the goal is to minimize the mean-squared-error between the predicted function value g and the actual function value x. Formally, we consider the following MLP:EQUATIONwhere σ is the activation of choice (ReLU, Tanh, or DIGRAF), andW1 ∈ R 2×64 , W2 ∈ R 64×64 , W3 ∈ R 64×1are the trainable parameter matrices of the linear layers in the MLP. The goal, as discussed above, is to minimize the loss ∥ĝ(x, y) -g(x, y)∥2, for data triplets (xi, yi, g(xi, yi)) sampled from the peaks function. In our experiment, we sample 50,000 points, and report the obtained approximation error in terms of MSE in Figure 6 . As can be seen, our DIGRAF, based on the CPAB framework, allows to obtain a significantly lower approximation error, up to 10 times lower (better) than ReLU, and 3 times better than Tanh. This example further motivates us to harness CPAB as the blueprint of DIGRAF. Our results are summarized in Table 5 , where we consider MUTAG, PTC, PROTEINS, NCI1 and NCI109 datasets from the TU repository [56] . As can be seen from the table, DIGRAF is consistently among the top-3 best-performing activation functions, and it consistently outperforms other graph-adaptive activation functions. These results support our design choices for DIGRAF and the flexibility offered by CPAB diffeomorphisms.We now provide a comparison between DIGRAF and the ReLU activation function coupled with GCN, GAT, GIN, and SAGE backbones in Table 6 . Notably, DIGRAF consistently outperforms ReLU regardless of the backbone architecture.To gain a qualitative understanding of the behavior of DIGRAF, we now illustrate the activation function learned by DIGRAF after the last GNN layer on different graphs. To this end, we randomly selected two graphs from the ZINC dataset, as shown in Figure 7 . The original graphs are presented in the lower right, with each color representing a feature. Nodes with the same color share the same feature. The comparison of the figures demonstrates that for different graphs, with different features and structures, DIGRAF learns distinct activation functions, showing its adaptivity to the input graph. The number of parameters is reported within the parenthesis adjacent to the metric. We use GINE [37] as a backbone. Increasing the parameter count with ReLU does not yield significant improvements, and DIGRAF outperforms all variants, even those with a higher number of parameters. Note that, DIGRAF (W/O ADAP.) has only N P -1 additional parameters, where N P is the tessellation size. GNNACT is a core component of DIGRAF, which ensures graph-adaptivity by generating the parameters θ (l) of the activation function conditioned on the input graph. While the benefits of graph-adaptive activation functions are evident from our experiments in Section 5, as DIGRAF consistently outperforms DIGRAF (W/O ADAP.), the variant of our method that is not graph adaptive, it comes at the cost of additional parameters to learn GNNACT (Equation ( 9)). Specifically, because in all our experiments GNNACT is composed of 2 layers and a hidden dimension of 64, DIGRAF adds at most approximately 20K additional parameters. The number of added parameters in DIGRAF (W/O ADAP.) is significantly lower, counting at NP -1, where NP is the tessellation size. Note in our experiments, the tessellation size does not exceed 16. To further understand whether the improved performance of DIGRAF is due to the increased number of parameters, we conduct an additional experiment using the ReLU activation function where we increase the number of parameters of the model and compare the performances. In particular, we consider following settings: (1) The standard variant (GIN + ReLU),The variant obtained by doubling the number of layers, and (3) The variant is obtained by doubling the number of hidden channels.We present the results of the experiment described above on the ZINC-12K and MOLHIV datasets in Table 7 .We observed that adding more parameters to the ReLU baseline does not produce significant performance improvements, even in cases where the baselines have ∼4 times more parameters than DIGRAFand its baseline. On the contrary, with DIGRAF significantly improved performance is obtained compared to the baselines.We present the impact of several key components of DIGRAF, namely the tessellation size NP , the depth of GNNACT (Equation ( 9)) and the regularization coefficient λ of θ (l) (Equation ( 12)). We choose a few representative datasets, i.e., CORA, CITESEER, FLICKR and BLOGCATALOG for which we use GCN [43] ; and ZINC-12K and MOLHIV for which we use GINE [37] as GNN respectively.Recall that the tessellation size NP determines the dimension of θ (l) ∈ R N P -1 that parameterizes the velocity fields within DIGRAF. We study the effect of the tessellation size on the performance of DIGRAF in Figure 8 . We can see that a small tessellation size is sufficient for good performance, and increasing its size results in marginal changes. This observation suggests that CPAB is highly flexible, and aligns with the conclusions in previous studies on different applications of CPAB [52] , which have shown that small sizes are sufficient in most cases.DIGRAF exhibits graph adaptivity by predicting θ (l) ∈ R N P -1 conditioned on the input graph through GNNACT. Table 8 shows the impact of the number of layers LACT of GNNACT on the performance of DIGRAF.In particular, we maintain a fixed architecture for DIGRAF and vary only LACT. The results show that increasing the depth of GNNACT improves only marginally the performance of DIGRAF, demonstrating that the increased number of parameters is not the main factor of the better performance of DIGRAF. On the contrary, the flexibility and adaptivity offered by DIGRAF are the main factors of the improvements, as demonstrated by DIGRAF consistently outperforming DIGRAF (W/O ADAP.) and other activation functions (Section 5).As discussed in Section 4.2, the regularization enforces the smoothness of the velocity field. We investigate the impact of the value of the regularization coefficient λ on DIGRAF (Equation ( 12)) in Table 9 . The results reveal that the optimal value of λ depends on the dataset of interest, with small positive values yielding generally good results across all datasets.To demonstrate the efficacy of graph adaptivity provided by GNNACT, we conduct an experiment where we increase the number of layers and channels of GNNLAYER in DIGRAF (W/O ADAP.) to match the total number of parameters in DIGRAF. As shown in Table 10 , the increase in the number of parameters does not translate to better performance. Rather, the effective usage of the extra parameters as done by GNNACT is the reason behind the performance boost offered by DIGRAF. We implemented DIGRAF using PyTorch [63] (offered under BSD-3 Clause license) and the PyTorch Geometric library [22] (offered under MIT license). All experiments were conducted on NVIDIA RTX A5000, NVIDIA GeForce RTX 4090, NVIDIA GeForce RTX 4070 Ti Super, NVIDIA GeForce GTX 1080 Ti, NVIDIA TITAN RTX and NVIDIA TITAN V GPUs. For hyperparameter tuning and model selection, we utilized the Weights and Biases (wandb) library [6] . We used the difw package [52, 25, 24, 74] (offered under MIT license) for the diffeomorphic transformations based on the closed-form integration of CPA velocity functions. In the following subsections, we present the experimental procedure, dataset details, and hyperparameter configurations for each task.Hyperparameters. The hyperparameters include the number of layers L and embedding dimension C of GNN (l) LAYER , learning rates and weight decay factors for both GNNLAYER and GNNACT, dropout rate p, tessellation size NP , and regularization coefficient λ. We additionally include the number of layers LACT and embedding dimension CACT of GNNACT. We employed a combination of grid search and Bayesian optimization. All hyperparameters were chosen according to the best validation metric. For the baselines, we include only the applicable hyperparameters in our search space.Node Classification. For each dataset, we train a 2-layer GCN [43] as the backbone architecture, and integrate each of the activation functions into this model. Following Zhang et al. [92] , we randomly choose 20 nodes from each class for training and select 1000 nodes for testing. For each activation function, we run the experiment 10 times with random partitions. We report the mean and standard deviation of node classification accuracy on the test set. Table 11 summarizes the statistics of the node classification datasets used in our experiments. All models were trained for 1000 epochs with a fixed batch size of 32 using the Adam optimizer. Tables 12 and 13 lists the hyperparameters and their search ranges or values.Graph Classification. The statistics of various datasets can be found in Table 14 . We consider the following setup:• ZINC-12K: We consider the splits provided in Dwivedi et al. [17] . We use the mean absolute error (MAE) both as the loss and evaluation metric and report the mean and standard deviation over the test set calculated using five different seeds. We use the Adam optimizer and decay the learning rate by 0.5 every 300 epochs, with a maximum of 1000 epochs. In all our experiments, we adhere to the 500K parameter budget [17] . We use GINE [37] layers both for GNN (l)LAYER and within GNNACT, and we fix CACT = 64 and LACT = 2. We report the hyperparameter search space for all the other hyperparameters in Table 15 .• TUDatasets: We follow the standard procedure prescribed in Xu et al. [84] for evaluation. That is, we use a 10 fold cross-validation and report the mean and standard deviation of the accuracy at the epoch that yields the best validation performance on average. We use the Adam optimizer and train for a maximum of 350 epochs. We use GIN [84] layers both for GNN (l)LAYER and within GNNACT, and we fix LACT = 2. We present the hyperparameter search space for all other parameters in Table 15 .• OGB: We consider 4 datasets from the OGB repository, with one, namely MOLESOL, being a regression problem, while the others are classification tasks. We run each experiment using five different seeds and report the mean and standard deviation of RMSE/ROC-AUC. We use the Adam optimizer, decaying the learning rate by a factor of 0.5 every 100 epochs, and train for a maximum of 500 epochs. We use the GINE model with the encoders prescribed in Hu et al. [36] both for GNN (l) LAYER and within GNNACT, and we set CACT = 64 and LACT = 2. We present the hyperparameter search space for all other parameters in Table 15 H Complexity and Runtimes Time Complexity. We now provide an analysis of the time complexity of DIGRAF. Let us recall the following details: (i) As described in Equation ( 8), DIGRAF is applied element-wise in parallel for each dimension of the output of GNN (l)LAYER . (ii) As described in Equation ( 9), we employ an additional GNN denoted by GNNACT to compute θ (l) . In all our experiments, both the backbone GNN and GNNACT are message-passing neural networks (MPNNs) [28] . (iii) As described in Theorem 2 of Freifeld et al. [25] , for 1-dimensional domain, there exists a closed form for T (l) (•; θ (l) ), and the complexity for the CPAB computations are linear with respect to the tesselation size, which is a constant of up to 16 in our experiments. Therefore, using DIGRAF with any linear complexity (with respect to the number of nodes and edges) MPNN Runtimes. Despite having linear computational complexity in the size of the graph, DIGRAF performs additional computations to obtain θ (l) using GNNACT. To understand the impact of these computations, we measured the training and inference times of DIGRAF and present it in Table 16 . Specifically, we report the average time per batch and standard deviation of the same measured on an NVIDIA A5000 GPU, using a batch size of 128. For a fair comparison, we use the same number of layers, batch size, and channels in all methods. Additionally, for our DIGRAF, we set the number of layers within GNNACT to LACT = 2, and the embedding dimension to CACT = 64. Our analysis indicates that while DIGRAF requires additional computational time, it yields significantly better performance. For example, compared to the best activation function on the dataset, namely Maxout, DIGRAF requires an additional ∼ 6.21ms at inference, but results in a relative improvement in the performance of ∼ 17.95%.On the ZINC dataset, using GIN as the primary model, DIGRAF exhibits approximately 4.5 times slower training times and 3.5 times slower inference times compared to ReLU. DIGRAF demonstrates an inference time that is approximately 1.35 times faster than GReLU, while also achieving superior performance.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 


Decision: The contribution of this paper consists of a novel activation function in learning by GNNs.  Specifically they introduce an approach, referred to as DIGRAF exploiting Continuous Piecewise-Affine Based (CPAB) transformations augmented with an additional GNN to learn a graph-adaptive diffeomorphic activation function in an end-to-end set up. In addition to  graph-adaptivity and flexibility, DIGRAFwas show to enjoy  desirable  properties for activation functions, including  differentiability, boundness. The authors have  run experiments to demonstrate a consistent and superior performance of DIGRAF relative to traditional and graph-specific activation functions.
All reviews but one who recommended weak accept, have recommended borderline accept. 
Most of the reviewers had recommended additional experiments (ablation, more baseline and other) and the authors have for the most part  complied and have provided additional experimental results.
The most negative review stated that the real  contribution, namely  the analysis  of the connection between CPAB and GNNs needed further theoretical exploration, which, while valid, raises the question of how much can be included in a given paper.
The second comment which is also a valid one is the non-inclusion of the code for the approach, which the authors respond to, by promising to publish the code after the paper is accepted, which is clearly enforceable.  
Overall the reviews were predominantly positive.

Reviews: review: paper_summary: paper_summary: The paper “DiGRAF: Diffeomorphic Graph Activation Functions” introduces DiGRAF, a novel graph neural network (GNN) activation function based on diffeomorphisms. DiGRAF adapts to graph structures and tasks by learning transformation parameters, enhancing performance across various GNN scenarios. The method demonstrates its effectiveness through extensive experiments on node classification, graph classification, and regression tasks, showing significant improvements over standard and graph-specific activation functions.
strength_weakness: strength_weakness: 1.      Limited Contribution: The paper appears to be more of an application of CPAB. The primary contribution of the analysis of the connection between CPAB and GNNs is not thoroughly explored.
2.	Performance and Gradient Optimization: The paper lacks a detailed analysis of performance and gradient optimization aspects.
3.	Clarification on GNN_act: It needs to be clearer how GNN_act helps boost performance. Is there a way to analyze the new source of information it brings?
4.	Proofs for GNN: Most analyses focus on the boundedness of the activation function without specifics tailored to GNNs, making it unclear why this approach is particularly suited for GNNs.
5.	Edge Classification Performance: The report on edge classification performance is missing.
6.  	Hyper-Parameter Effects: It is not clear what the effect of each hyper-parameter is on the performance of the activation function.
7.   	Source Code: The source code is not available.
questions: questions: How are the parameters {b_j} (orthonormal basis of the space of velocity fields V) calculated?
2.	What would be the activation function of GNN_act if the method’s purpose was to design a new activation function?
3.	Are there results for other GNN structures beyond GCN? For instance, GReLU reports results for different GNN structures.
4.	Since DiGRAF is also applicable to non-graph datasets, would it be beneficial to see its results on other machine learning methods?
5.	What is the closed-form solution of the integral in Equation (2) for 1-dimensional vectors?
6.	Is there any information on the learned \theta parameters in terms of sparsity or graph structure?
limitations: limitations: Inductive Learning: Since the method is task-dependent, it loses its inductive capabilities and cannot be used for cases it did not see.
2.	Hyperparameter Choice: There is a dependency on hyperparameter tuning, which may affect generalizability.
3.	Time Complexity: DiGRAF has higher time complexity compared to standard activation functions.
4.	Decentralized Tasks: The approach is not ideal for decentralized tasks where there is no central data.
paper_summary: paper_summary: This paper introduces DIGRAF, Diffeomorphic Graph-Adaptive Activation Function, which is a novel activation function adaptive to graph data. DIGRAF leverages Continuous Piecewise-Affine Based transformations, possesses several necessary properties of a good activation function, such as differentiability, zero-centering and permutation equivariance. The author allows DIGRAF adapt to graph data by using an additional GNN to learn the final activation function formula. Adequate experiments which take use of DIGRAF in GNNs proved the high efficiency of this activation function.
strength_weakness: strength_weakness: There should be an ablation study (can be set in the supplementary part) about the hyper-parameters and training strategy in the process of leaning DIGRAF.
questions: questions: 1. In the experiments, why was the sigmoid function not used as an activation function?
2. Are there any differences in the DIGRAF learning process for GCN and GIN?
limitations: limitations: 1. As the author mentioned, the current DIGRAF may not be optimal since it was learned only for simple tasks, but it can be improved in future work.
2. The application of DIGRAF has not been discussed or implemented in this research. However, using DIGRAF as a novel activation function in some current models might improve their efficiency.
paper_summary: paper_summary: This paper introduces a new activation function, DIGRAF, specifically tailored for graph data in Graph Neural Networks (GNNs). The approach is based on Continuous Piecewise-Affine Based transformation (CPAB). The authors demonstrate that DIGRAF possesses the desired properties highlighted in existing literature and provide a thorough analysis of these properties. Extensive experiments conducted on diverse datasets across various tasks show that DIGRAF outperforms three different types of baselines in downstream performance.
strength_weakness: strength_weakness: 1. Apart from the adoption of the CPAB approach, the primary contribution appears to be the introduction of GNN_act. However, the discussion on why GNN_act is effective is relatively brief. Including more detailed discussions or theoretical justifications would help in understanding its advantages.
2. How do the properties of DIGRAF influence the convergence? It would be nice if it could be discussed which property the gain comes from.
3. As I mentioned above, more ablation studies should be done to reveal how each property and design contribute to the performance gain. Otherwise, DIGRAF (w/o adap.) is good and simple enough. Maybe we can save the budget of GNN_act for a larger GNN_layer.
questions: questions: 1. The size of GNN_act seems to have a minor influence (non-monotonic) on performance according to Appendix F.2. What if you increase the size of GNN_layer a little bit (add the parameter of GNN_act to GNN_layer) and compare DIGRAF (w/o adap. & larger GNN_layer) with the current DIGRAF? I believe this could provide a better justification for the effectiveness of GNN_act. 
2. What are the properties that DIGRAF has but general-purpose activation functions or existing graph activation functions do not have? Can you put up a table to clarify it?
3. How do the properties of DIGRAF influence the convergence and performance? 
4. The tessellation size decides the degree of freedom of DIGRAF, right?  Why does increasing the tessellation size result in marginal changes (non-monotonic)?
limitations: limitations: The authors have discussed the limitations.
paper_summary: paper_summary: Inspired by the continuous piecewise-affine based transformation, this paper argued that the activation function should not be an uniform selection for different nodes, and developed a learning activation function method, that can take the graph structure into account to activate node-specific features. The experimental results support their proposed model's effectiveness through comparing with existing activation functions in several datasets.
strength_weakness: strength_weakness: Many formula seem to be repeated, such as equations in 3.2 and 4.1.
The complexity is a bit high compared with an existing activation function.
questions: questions: 1. How will you ensure the parameter \theta you learn from the training can return you a diffeomorphism activation function? It is not clearly stated in your experimental settings, such as how you define transformation T and space \Omega.
2. The method used a GNN to learn the activation function, is there an activation function in the GNN_act?
3. In the node classification experiments, there are some significant improvement in some tasks, while some of them have a minor improvement. Have you tried to figure out the reasons?
limitations: limitations: There is no potential negative societal impact of this work.

Comments: comment: Thanks for your response. I read it carefully.
Dear Reviewers,

We thank you for the effort in reviewing our paper, and the overall positive feedback. \
We are also thankful for the insightful questions, comments, and suggestions, that helped us improve the paper. 

As the rebuttal period ends soon, we wanted to reach out and check if you feel that our rebuttal answers your questions. \
We have worked to thoroughly address your comments, and our rebuttal includes clarifications and discussion to important comments raised by the reviewers, as well as additional experimental results as described in our General Response.\
All of the additional results and discussions were also added to our revised paper.

Please, kindly let us know of any further questions you might have. We look forward to the discussion with you.

Thank you, and best regards,\
Authors.