SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this experiment, we generate a random graph  $\mathcal{G} = \langle \mathcal{V},\mathcal{E}\rangle$  with  $|\mathcal{V}| = 8$  nodes and an edge density of  $\frac{|\mathcal{E}|}{|\mathcal{V}|^2} = 0.3$ . For each node  $i\in \mathcal{V}$ , we draw an internal state  $x_{i}\in \mathbb{R}^{d}$  from a normal distribution  $x_{i}\sim \mathcal{N}(\mathbf{0}_{d},I_{d})$  with  $d = 6$ . Then, we generate training data with a set of ground truth aggregators  $\odot_k\in \mathcal{A}$  (where  $k$  is an index). For each aggregator  $\odot_k$ , the dataset  $X_{k},Y_{k}$  is produced with a Graph Network [2], using  $\odot_k$  as the node aggregation module. The inputs are defined by the set of neighbourhoods in the graph  $X_{k} = \{\mathcal{X}_{i}\mid i\in [1..|\mathcal{V}|]\}$  where the neighbourhood  $\mathcal{X}_i$  is defined as  $\mathcal{X}_i = \{x_j\mid j\in \mathcal{N}_i\}$  with  $\mathcal{N}_i = \{j\mid (i,j)\in \mathcal{E}\}$ . The corresponding ground truth outputs are defined as  $Y_{k} = \{y_{i}\mid i\in [1..|\mathcal{V}|]\}$ , where  $y_{i} = \odot_{k}(\mathcal{X}_{i})$ .
The model that we use for regression takes the same form as the model used to generate the data, except that the standard aggregator used to generate the training data  $\odot_{k}$  is replaced with a parametrised aggregator  $\bigoplus_{\theta}$ :
$$
\hat {y} _ {i} = \bigoplus_ {x _ {j} \in \mathcal {X} _ {i}} x _ {j} \tag {4}
$$
In our experiments, each type of parametrised aggregator (GenAgg, SoftmaxAgg, PowerAgg, and mean as a baseline) is trained separately on each dataset  $X_{k}, Y_{k}$ .
Results. We report the MSE loss and correlation coefficient with respect to the ground truth in Table 2a. GenAgg is able to represent all of the standard aggregators with a correlation of at least 0.96, and most aggregators with a correlation of greater than 0.99. The only cases where the performance of GenAgg is surpassed by a baseline are min and max, where SoftmaxAgg exhibits marginally higher accuracy.
One interesting observation is that even if the baselines can represent an aggregator in theory, they cannot necessarily do so in practice. For example, PowerAgg can theoretically represent the geomet-
Figure 3: Test accuracy for GNNs with various aggregators on GNN benchmark datasets. In this experiment, each trial uses the same base GNN architecture (4-layer GraphConv), and the default aggregator is replaced with either GenAgg, PowermeanAgg (P-Agg), SoftmaxAgg (S-Agg), PNA, mean, sum, or max. The plots depict the mean and standard deviation of the test accuracy over 10 trials (note that the y-axis is scaled to increase readability). The table reports the maximum of the mean test accuracy over all timesteps, as well as the standard deviation (rounded to 0.01).
Dataset	GenAgg	P-Agg	S-Agg	PNA	mean	sum	max
CIFAR10	0.540	0.451	0.467	0.473	0.502	0.435	0.471
±0.09	±0.14	±0.13	±0.14	±0.13	±0.15	±0.13
MNIST	0.928	0.877	0.864	0.717	0.847	0.844	0.831
±0.07	±0.07	±0.06	±0.19	±0.08	±0.08	±0.08
CLUSTER	0.627	0.610	0.611	0.168	0.602	0.170	0.501
±0.02	±0.01	±0.01	±0.01	±0.01	±0.02	±0.01
PATTERN	0.925	0.883	0.896	0.861	0.871	0.860	0.860
±0.00	±0.01	±0.03	±0.01	±0.01	±0.01	±0.01
ric mean with  $\lim_{p\to 0}\left(\frac{1}{n}\sum_{i}x_{i}^{p}\right)^{\frac{1}{p}}$ , but in practice there are instabilities as  $p$  approaches 0 because  $\frac{1}{p}$  approaches  $\frac{1}{0}$ . Similarly, while in theory PowerAgg can represent min magnitude, max magnitude, harmonic mean, and root mean square, it falls short in practice (see Table 2a), likely because of the reasons stated in Section 4. In other cases, the baselines can perform well even if they should not be able to represent the target in theory. One such example is PowerAgg, which achieves a correlation of 0.92 on max, but only 0.59 on max magnitude, which is the opposite of what theory might suggest. This is likely due to the clamp operation that Pytorch Geometric's implementation uses to restrict inputs to the positive domain. The performance of max magnitude suffers, as it misses cases where the highest magnitude element is negative. Similarly, the performance of max increases, because it simply selects the maximum among the positive elements. Another baseline which performs unexpectedly well is SoftmaxAgg, which achieves a high correlation with the log-sum-exp aggregator. While it cannot compute a log, the SoftmaxAgg formulation does include a sum of exponentials, so it is able to produce a close approximation.
5.2 GNN Regression
In GNN Regression, the experimental setup is the same as that of Aggregator Regression (Section 5.1), with the exception that the observation size is reduced to  $d = 1$ . However, instead of using GenAgg  $\bigoplus_{\theta}$  as a model, we use a multi-layer GNN. The GNN is implemented with 4 layers of GraphConv [16] with Mish activation (after every layer except the last), where the default aggregation
tion function is replaced by a parametrised aggregator  $\bigoplus_{\theta}$
$$
z _ {i} ^ {(k + 1)} = \operatorname {M i s h} \left(W _ {1} ^ {(k)} z _ {i} ^ {(k)} + W _ {2} ^ {(k)} \bigoplus_ {j \in \mathcal {N} _ {i}} z _ {i} ^ {(k)}\right), \text {s . t .} z _ {i} ^ {(0)} = x _ {i} \tag {5}
$$
$$
\hat {y} _ {i} \quad = W _ {1} ^ {(3)} z _ {i} ^ {(3)} + W _ {2} ^ {(3)} \bigoplus_ {j \in \mathcal {N} _ {i}} z _ {i} ^ {(3)} \tag {6}
$$
While this experiment uses the same dataset as Aggregator Regression (Section 5.1), it provides several new insights. First, while the aggregator regression experiment shows that GenAgg can represent various aggregators, this experiment demonstrates that training remains stable even when used within a larger architecture. Second, this experiment underlines the importance of using the correct aggregation function. While it is clear that it is advantageous to match a model's aggregation function with that of the underlying mechanism which generated a particular dataset, we often opt to simply use a default aggregator. The conventional wisdom of this choice is that the other learnable parameters in a network layer can rectify an inaccurate choice in aggregator. However, the results from this experiment demonstrate that even with additional parameters, it is not necessarily possible to represent a different aggregator, underlining the importance of aggregators with sufficient representational capacity.
Results. The results show that GenAgg maintains its performance, even when used as a component within a GNN (Table 2b). GenAgg achieves a mean correlation of 0.97 across all aggregators. While the baselines perform significantly better with the help of a multi-layer GNN architecture, they still cannot represent many of the standard aggregators. The highest-performing baseline is SoftmaxAgg, which only achieves a mean correlation of 0.86.
5.3 GNN Benchmark
In this experiment, we examine the performance of GenAgg on GNN benchmark datasets [6]. In order to perform a comparison with benchmarks, we train on an existing GNN architecture (a 4-layer GraphConv [16] GNN with a hidden size of 64) where the default aggregator is replaced with a new aggregator, selected from  $\{\mathrm{GenAgg},\mathrm{PowerAgg},\mathrm{SoftmaxAgg},\mathrm{PNA},\mathrm{mean},\mathrm{sum},\mathrm{max}\}$ .
Results. As shown in Fig 3, GenAgg outperforms all baselines in all four GNN benchmark datasets. It provides a significant boost in performance, particularly compared to the relatively small differences in performance between the baseline methods.
The training plots in Fig. 3 provide complementary information. One interesting observation is that GenAgg converges at least as fast as the other methods, and sometimes converges significantly faster (in PATTERN, for example). Furthermore, the training plots lend information about the stability of training. For example, note that in MNIST, most of the baseline methods achieve a maximum and then degrade in performance, while GenAgg maintains a stable performance throughout training.
Results. In our experiments, we present two regression tasks and one GNN benchmark task. The regression experiments demonstrate that GenAgg is the only method capable of representing all of the standard aggregators, and a GNN cannot be used to compensate for the shortcomings of the baseline aggregators. The GNN benchmark experiment complements these findings, demonstrating that this representational complexity is actually useful in practice. The fact that GenAgg outperforms the standard aggregators (mean, max, and sum) on the GNN benchmark experiment implies that it is in fact creating a new aggregator. Furthermore, the fact that it outperforms baseline methods like SoftmaxAgg and PowermeanAgg implies that the aggregator learned by GenAgg lies outside the set of functions which can be represented by such methods.
Limitations. While GenAgg achieves positive results on these datasets, it is not possible to make generalisations about its performance in all applications. In particular, we observe that some datasets fundamentally require less complexity to solve, so simple aggregators are sufficient (i.e., GenAgg fails to provide a significant performance boost). For a full list of datasets that we considered and further discussion of limitations, see Appendix D.
Parameters. When comparing the performance of different models, it is important to also consider the number of parameters. By introducing additional parameters, some models can improve overall performance at the cost of sample efficiency. While methods like PowerAgg and SoftmaxAgg only have one trainable parameter, GenAgg has two scalar parameters  $\alpha$  and  $\beta$ , and a learnable function  $f$ , which has 30 parameters in our implementation (independent of the size of the state). However, we observe that using GenAgg within a GNN is always at least as sample-efficient as the baselines, and sometimes converges significantly faster (Fig. 3 and Appendix B). Furthermore, while GenAgg has more parameters than PowerAgg and SoftmaxAgg, the increase is negligible compared to the total number of parameters in the GNN. We also note that GenAgg has significantly fewer parameters than the deep learning methods discussed in Section 4. While the deep learning methods scale linearly or quadratically in the dimension of the state, the number of parameters in GenAgg is constant.
Stability. Another observation from our experiments is that GenAgg exhibits more stability during the training process than the baselines (Appendix B). In the GNN Regression experiment, the PowerAgg and SoftmaxAgg training curves tend to plateau at least once before reaching their maximum value. It is possible that these methods lead to local optima because they are optimised in a lower dimensional parameter space [3]. For example, it is straightforward to smoothly transform a learned  $f$  in GenAgg from  $x^{2}$  to  $x^{4}$ , but to do so in PowerAgg, it is necessary to pass through  $x^{3}$ , which has significantly different behaviour in the negative domain. While PowerAgg restricts inputs to the positive domain to circumvent this particular issue, the problem of local optima can still arise when methods like PowerAgg or SoftmaxAgg are used as components in a larger architecture.
Explainability. While in this paper we primarily focus on the performance of GenAgg, we note that it also presents benefits in the realm of explainability. The three parameters in GenAgg are all human-readable (calvars and scalar-valued functions can easily be visualised), and they all provide a unique intuition. The  $\alpha$  parameter controls the dependence on the cardinality of the input set. The  $\beta$  parameter dictates if the aggregator is computed in a raw or centralised fashion (colloquially, it answers if the aggregator operates over the inputs themselves, or the variation between the inputs). Lastly, the function  $f$  can be analysed by considering the sign and magnitude of  $f(x_{i})$ . The sign denotes if a given  $x_{i}$  increases  $(f(x) > 0)$  or decreases  $(f(x_{i}) < 0)$  the output. On the other hand, the magnitude  $|f(x_{i})|$  can be interpreted as the relative impact of that point on the output. For example, the parametrisation of product is  $f(x) = \log(|x|)$ , which implies that a value of 1 has no impact on the output since  $|\log(|1|)| = 0$ , and extremely small values  $\epsilon$  have a large impact, because  $\lim_{\epsilon \to 0} |\log(|\epsilon|)| = \infty$ . Indeed, 1 is the identity element under multiplication, and multiplying by a small value  $\epsilon$  can change the output by many orders of magnitude. The interpretability of GenAgg can also be leveraged as a method to select an aggregator—a model can be pre-trained with GenAgg, and then each instance of GenAgg can be replaced with the most similar standard aggregator in  $\mathcal{A}$ .
In this paper we introduced GenAgg, a generalised, explainable aggregation function which parametrises the function space of aggregators, yet remains as constrained as possible to improve sample efficiency and prevent overfitting. In our experiments, we showed that GenAgg can represent all 13 of our selected "standard aggregators" with a correlation coefficient of at least 0.96. We also evaluated GenAgg alongside baseline methods within a GNN, illustrating how other approaches have difficulties representing standard aggregators, even with the help of additional learnable parameters. Finally, we demonstrated the usefulness of GenAgg on GNN benchmark tasks, comparing the performance of the same GNN with various different aggregators. The results showed that GenAgg provided a significant boost in performance over the baselines in all four datasets. Furthermore, GenAgg often exhibited more stability and faster convergence than the baselines in the training process. These results show that GenAgg is an application-agnostic aggregation method that can provide a boost in performance as a drop-in replacement for existing aggregators.
Ryan Kortvelesy and Amanda Prorok were supported in part by ARL DCIST CRA W911NF-17-2-0181 and European Research Council (ERC) Project 949940 (gAla).
[23] Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.