Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: (**Disclaimer**: I was assigned to this paper as an emergency reviewer;
sorry for being slightly more terse than I would like to be ideally)

This paper presents an empirical analysis of the predictive performance
of graph neural networks when switching to different readout functions.
A set of such readout functions is investigated, some of which do *not*
satisfy permutation invariance by design. Surprisingly, the performance
of such readout functions turns out to be beneficial on some graph data
sets and some learning tasks!

strengths_and_weaknesses: The main strength of the paper lies in the strong experimental section,
which challenges the status quo. The use of pooling functions that satisfy 
permutation invariance has been deeply ingrained in the graph learning
literature. Thus, an investigation of their efficacy in practice is
highly appreciated and useful.

The main weakness of the paper is a missing discussion about parameter
budgets; to a certain extent, some of the observed differences could
potentially also be explained by having 'more expressive' readout
functions (permutation invariance notwithstanding). This should be
clarified but can easily be accomplished within the conference reviewing
cycles (see questions below).

Moreover, a more specific discussion of 'takeaway messages' could be
added, making some recommendations for practitioners in light of the
observed phenomena.

In summary, I believe that this paper requires only minor modifications
before being ready for publication. It will make an excellent
contribution to the conference.

## Detailed Comments

- When discussing graph expressivity, I would suggest to rather state
  that certain certain GNNs are *not more expressive than the WL test*,
  i.e. a 'shallow' method without trainable weights. Moreover, the test
  is not as computationally efficient as GNNs; this should be rectified.

- The beginning of Section 2 could potentially be condensed to make
  more space for experiments or discussions, which are critical for this
  type of paper.

- The fact that differences are statistically significant (Appendix R)
  should be mentioned more prominently in the main text in Figure 1.

- Figure 1 could potentially also be improved by showing standard
  deviations of the 'best-to-best ratio'.

- Figure 2 (and other figures as well) should use a consistent colour
  scheme and ensure that the *new* readout functions are highlighted
  somewhat prominently.

- More information about the dimensionality of output vectors should be
  provided, in particular when discussing aspects like convergence or
  Euclidean distance (see also next point).

- For the calculation of Euclidean distances (Appendix I), more details
  about the models and the dimensionalities should be provided. It is
  possible (though not likely) that the changes in distances can also be
  explained by some 'nuisance' dimensions in the outputs. Moreover, this
  type of analysis would lend itself well to a repetition in order to
  get an idea of the variance in the data.

## Minor Comments

- Eq. (3) could be improved; there are several terms to keep track of at
  the moment; $\mathrm{SAB}$ could for instance just be substituted by
  $\mathrm{MAB}$.

- Space permitting, a delineation to the results from the Janossy
  pooling paper could be provided. Having read both papers, I recall
  that Janossy pooling *also* discusses the merits of a different
  readout function.

- Why are some $p$-values zero in the appendix? Is this a problem of
  typesetting the table?

- Please check the bibliography again; there are some citations that
  should refer to published work but point to a preprint at the moment
  (for example, Alon and Yahav [20] is now published at ICLR; the work
  by Kipf and Welling [1] was likewise published at ICLR).

- When discussing WL, it might be interesting to cite a [recent survey
  that summarises the connections of WL to machine learning](https://arxiv.org/abs/2112.09992).

questions: 1. Is there a reason why MCC was used instead of accuracy differences or
   accuracy ratios?

2. To what extent could the observed performance increases by explained
   by having more parameters available for a model to fit? I am
   mentioning this because it appears that one recommendation is to use
   a set transformer, i.e. a permutation-invariant readout function.
   This slightly changes the message of the paper to 'More parameters
   for readout functions are better,' whereas initially, the goal was to
   see whether permutation invariance is actually needed. Please comment
   on this.
limitations: Limitations are not directly addressed. A brief statement on the broader
impact of the work would be welcome. At the same time, I do not foresee
any adverse outcomes that arise specifically because of this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces novel adaptive and differentiable readout functions for Graph Convolutional Networks that rely on neural networks and which do not necessarily result in permutation invariant hypothesis space.
The authors provide an extensive empirical evaluation on 40 datasets which involves some standard readout/pooling policies and different neighborhood aggregation schemes.

strengths_and_weaknesses: Strengths:
- Several datasets and different domains considered in the experiments
- The authors empirically analyze several different aspects, including an ablation study that highlights the impact of various parts of the model.

Weaknesses:
- Crucial parts of the results are reported only in the appendix
- Some aspects of the empirical evaluation have to be clarified (e.g validation strategy)
- Missing some references
- Some common graph pooling methods are not considered in the experimental comparison
questions: It is not clear why some pooling techniques like the one reported in [19], [20[ [21], and [22] were not considered in the empirical evaluation.

In section 3 (rows 200-201) the authors state “Thus, it200 is likely that the performance can be further improved by hyperparameter tuning.”, therefore I wonder whether (and how) the authors performed a validation phase and how the hyper parameters used in the various experiments were selected.

In the subsection “Neural vs standard readouts across different datasets and neighborhood aggregation schemes'' the authors use two-layer GNN. This choice clearly introduces a bias in the valuation strategy. How much does it impact the obtained results?
limitations: The paper turns out to be very interesting and I appreciate the considerable amount of comparisons and evaluation that the authors performed to assess the proposed approach. The point is that all the reported analyses and evaluations require more than a 9-page conference paper to be described and discussed. Indeed the authors report a considerable quantity of the results (crucial to understanding the proposed discussion) and a part of the description of the architecture in the appendix. In my opinion that makes the paper difficult to read. The organization of the information and its division between the main paper and appendix have to be revised.

Some references were missing, e.g the wide use of graph-level aggregators like DiffPool (Ying, R., You, J., Morris, C., Ren, X., Hamilton, W.L., Leskovec, J.: Hierarchical graph representation learning with differentiable pooling (2019)) and the Sortpool (Zhang, M., Cui, Z., Neumann, M., Chen, Y.: An end-to-end deep learning architecture for graph classification. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018))  that have to be at least discussed. Same for the recently proposed SOM-based aggregator (Pasa, L., Navarin, N., Sperduti, A.: Som-based aggregation for graph convolutional neural networks. Neural Computing and Applications pp. 1–20 (2020)).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper provides a comprehensive empirical study of graph neural networks (GNN) with adaptive readouts. Graph neural networks traditionally use non-adaptive readouts based on simple functions (like sum, max, mean) to preserve permutation invariance. Experiments in this paper demonstrate that permutation invariance can be traded with performance improvement by using learnable, adaptive readout functions based on neural architectures. 
strengths_and_weaknesses: Strengths:
This paper combines traditional GNN architectures with three adaptive readouts in its empirical evaluations and therefore, there are no significant technical advancements. The main strength of the paper is the comprehensive empirical analysis for datasets that span multiple domains (e.g. bioinformatics, computer vision, quantum mechanics etc.) from the existing literature. The experiments demonstrate considerable improvements in performance using adaptive readouts over that for standard readout functions. The experiments are well explained and the accompanying discussions are illuminating from an applied machine learning perspective. 


Weaknesses: 
1. I think the property of transferability of GNNs (see ref. [a] below) merits significant discussion in the paper. Adopting adaptive readouts in GNNs affects transferability in conjunction with permutation invariance.
2. The focus of the paper is on improvement in performance using adaptive readouts. However, in some applications, the interpretability of the results can be equally or more important than performance. An example of this scenario is brain age prediction, where the error in brain age may have prognostic value and potentially predict health outcomes in disease (see ref. [b] below). 


[a] Ruiz, Luana, Luiz Chamon, and Alejandro Ribeiro. "Graphon neural networks and the transferability of graph neural networks." Advances in Neural Information Processing Systems 33 (2020): 1702-1712.

[b] Lee, J., Burkett, B.J., Min, HK. et al. Deep learning-based brain age prediction in normal aging and dementia. Nat Aging 2, 412–424 (2022). https://doi.org/10.1038/s43587-022-00219-7

questions: 1. A brief description of Matthew correlation coefficient would improve readability.
2. The bio-affinity prediction task is not clear to me and I believe some details are missing. For instance, what is SMILES representation in this context? I suggest adding appropriate details or references to explain this task. 
3. Can the authors comment on the relevance of neighborhood aggregations with a multilayer perceptron based DENSE readout layer? In this case, I am concerned that neighborhood aggregations might become irrelevant with the learning performance primarily attributable to the DENSE layer for at least some of the experiments. 
4. I recommend adding discussions on transferability of GNNs with adaptive readouts. 
limitations: Yes.
review_title: null

Meta-review: metareview: The paper proposes the use of an adaptive readout function in GNNs together with extensive empirical work to support it. The reviewers all found the paper interesting and are generally in favor of accepting it (with one marking strong accept with high confidence). Therefore, I recommend the paper be accepted, and encourage the authors to take into account the reviewer comments (as they have also indicated in their responses) when preparing the camera ready version. In particular, I would like to encourage them to use the extra page given there to reconsider the split of materials between main paper and appendix.
confidence: Certain
Decision: Accept