Question: Does GNP have a higher complexity or more parameters than baselines?
Evidence: The GNP function in equation 3.1 is not continuous and even not defined at p = 0. Hence, directly learning GNP in the original form as in equation 3.1 (even with p = 0 ignored) can cause instability, especially when an algorithm is trying to move from a positive p value to a negative p value. Instead, we suggest splitting the GNP function into two parts, GNP + with positive p and GNP − with negative p, and let the model choose the right balance between them. Specifically, define
where p + > 0, q + , p − > 0, and q − are learnable parameters. Given a set of node features V, we first split the feature dimension into two, and compute the output from GNP + for the first half and from GNP − for the second half. Then we mix two outputs with a single linear layer to get the final output.
where • is the floor function, W ∈ R d×d and b ∈ R d are learnable parameters. Note that widelyused GNN layers have a linear layer or MLP after message-passing between nodes. Instead of using an additional linear layer, GNP concatenates the outputs of GNP + and GNP − and passes them to the linear layer or MLP. Therefore, we have only four extra parameters (p + , p − , q + and q − ) for each GNN layer of typical GNN architectures. With this design, GNP can easily switch between positive p and negative p, choosing proper values according to tasks.
Many real-world data, such as relationships between people in social networks or chemical bonds between atoms, can naturally be represented as graphs. Finding models with proper inductive biases to better describe such graph data has been a common goal for many researchers, and Graph Neural Networks (GNNs) are considered to be the most successful model. They have proved effective for a variety of tasks, including recommendation, drug discovery, and chip design.
An important design choice for a GNN often overlooked is the specification of pooling functions, the functions used for the aggregation or readout operation in GNNs. They are usually required to be invariant w.r.t. the permutation of nodes in a graph, and common choices are element-wise summation (sum), maximum (max), minimum (min), or average (mean). Some recent works also proposed to use parametric models and learn them from data as well.
While most of the previous works on this line focused on improving predictive performance for their own tasks, recently, studied the impact of the choice of pooling functions on the ability of a neural network to extrapolate. Specifically, highlighted the importance of the choice of pooling functions in order to make GNNs generalize over the data lying outside of the support of the training data distribution, and they argued that the specification of the pooling functions acts as an important inductive bias that can make GNNs either completely fail to extrapolate or gracefully generalize to out-of-distribution data. As a motivating example, consider the problem of counting the number of nodes in a graph. If we are to solve this problem with a single-layer GNN having one readout layer, probably the best pooling function would be sum, and the corresponding model will readily generalize to graphs with a much larger number of nodes than the ones seen during training. On the other hand, if we choose the pooling function as max instead, it may still fit the training data well but completely fail to predict the number of nodes in out-of-distribution graphs.
The findings in raise a natural question; which pooling functions should be used for a given problem in order to make GNNs constructed with them successfully extrapolate for out-of-distribution data? did not present any guide but empirically showed that we do have the "right" pooling function for each problem tested, and when a pooling function is not properly selected, GNNs completely fails to extrapolate. The caveat here is that we do not know which pooling function is the right choice before actually training and validating the model.
To this end, in this paper, we present a generic learning-based method to find proper pooling functions for a given arbitrary problem. Our method, entitled Generalized Norm-based Pooling (GNP), formulates the pooling functions as a generic L p norm-like function (including negative p as well), and learns the parameters inside the pooling functions in an end-to-end fashion. Unlike previous learning-based pooling methods that are usually tailored for specific tasks or focused on improving predictive performances, GNP can be applied to arbitrary tasks, and it improves the extrapolation ability of GNNs constructed with it. Also, GNP includes most of the pooling functions being used for GNNs as special cases. Despite the enhanced flexibility, GNP incurs minimal overhead in GNN in terms of the model complexity. A naïve application of GNP to GNNs is likely to fail because of some difficulty in training, so we propose a simple remedy to this. Using nine graph-level, node-level, and set-related tasks, we demonstrate that GNNs with GNP trained by our training scheme extrapolate for out-of-distribution data comparably and sometimes even better than those with pooling functions that are carefully chosen among of widely-used ones. In addition, we demonstrate the effectiveness of GNP on four real-world tasks (graph classification, influence maximization, node classification, and graph regression) using six additional GNN architectures, GAT, PNA, hierarchical SAGPool, ASAPool, and MONSTOR) and eight real-world graphs. We summarize our contributions as follows:
• Generalized pooling function: We propose GNP, a simple yet flexible pooling function that can readily be applied to arbitrary tasks involving GNNs, with minimal parameter overhead. • Effective training methods: We propose effective training methods for GNP.
• Extensive experiments: We empirically demonstrate that GNNs with GNP generalize to out-ofdistribution data on nine extrapolation tasks. We also show successful application of GNP to six GNN architectures on four real-world tasks in eight real-world graphs with up to one million edges.
Instead of using an additional linear layer, GNP concatenates the outputs of GNP+ and GNP− and passes them to the linear layer or MLP. Therefore, we have only four extra parameters (p +, p −, q + and q −) for each GNN layer of typical GNN architectures.****In addition, we demonstrate the effectiveness of GNP on four real-world tasks (graph classification, influence maximization, node classification, and graph regression) using six additional GNN architectures (GCN (Kipf & Welling, 2017), GAT (Velickovic et al., 2018), PNA (Corso et al., 2020), hierarchical SAGPool (Lee et al., 2019b), ASAPool (Ranjan et al., 2020), and MONSTOR (Ko et al., 2020)) and eight real-world graphs.
Answer: We have only **four** extra parameters ((p +, p −, q + and q −) for each layer, compared to other widely-used GNN layers such as GCN and GIN. Note that they also have a linear layer or MLP after message-passing between nodes. Instead of using an additional linear layer, GNP concatenates the outputs of GNP+ and GNP− and passes them to the linear layer or MLP. Also, note that we set the dimensions so that the concatenated output of GNP has the same dimension as the output of other GNNs. These facts are clarified in the revised paper.