Reproducibility and Geometric Intrinsic Dimensionality: An Investigation on Graph Neural Network Research.
Authors: Tobias Hille
Maximilian Stubbemann
Tom Hanika
Venue: Accepted by TMLR ()

Abstract: Difficulties in replication and reproducibility of empirical evidences in machine learning
research have become a prominent topic in recent years. Ensuring that machine learning 
research results are sound and reliable requires reproducibility, which verifies the reliability of
research findings using the same code and data. This promotes open and accessible research,
robust experimental workflows, and the rapid integration of new findings. Evaluating the
degree to which research publications support these different aspects of reproducibility is
one goal of the present work. In order to do this, we introduce an ontology of reproducibility
in machine learning and apply it to methods for graph neural networks.
The objective of this study is to try and identify hidden effects that influence model 
performance. To this end, we employ the aforementioned ontology to control for a broad 
selection of sources and turn our attention to another critical challenge in machine learning. 
The curse of dimensionality, which induces complication in data collection, representation, 
and analysis, makes it harder to find representative data and impedes the training and inference
processes. The closely linked concept of geometric intrinsic dimension is employed to 
investigate the extent to which the machine learning models under consideration are influenced
by the intrinsic dimension of the data sets on which they are trained.

Published in Transactions on Machine Learning Research (10/2024)
Reproducibility and Geometric Intrinsic Dimensionality: An Investigation on Graph Neural Network Research.
Tobias Hille@*
hille uni-kossel
Knowledge & Data Engineering Group. University of Kassel, Kassel, Germany
Interdisciplinary Research Center for Information System Design, University of Kassel, Kassel, Germany
Maximilian Stubbemann
stabbemannShimmll.de
Information Systems and Machine Learning Lab, University Hildesheim, Hildesheim, Germany
Knowledge & Dota Engineering Group, University of Kassel, Kassel, Germany Interdisciplinory Research Center for Information System Design, University of Kassel, Kassel, Germany
Tom Hanika
Intelligent Information Systems, University of Hildesheim, Hildesheim, Germang Knowledge & Data Engincering Group, University of Kassel, Kassel, Germany Interdisciplinory Research Center for Information System Design, University of Kassel, Kassel, Germany
Reviewed on OpenReview: https: openreview. net/ forum? id=CtEGaIqtud
Abstract
Difficulties in replication and reproducibility of empirical evidences in machine learning research have become a prominent topic recent years. Ensuring that machine learning ro- search results sound and reliable requires reprodueibility, which verifies the reliability of research findings using the same code and data. This promotes open and accessible research, robust experimental workflows, and the rapid integration of new findings. Evaluating the degree to which research publications support these different aspects of reproducibility is one goal of the present work. In order to do this, we introduce an ontology of reproducibility in machine learning and apply it to methods for graph neural networks.
The objective of this study is try and identify hidden effects that influence model perfor- mance. To this end, we employ the aforementioned ontology to control for broad selection of sources and turn our attention to another critical challenge in machine learning. The curse of dimensionality, which induces complication in data collection, representation, and analysis, makes it harder to find representative data and impedes the training and inference processes. The closely linked concept of geometric intrinsic dimension is employed to inves- tigate the extent to which the machine learning models under consideration are influenced by the intrinsic dimension the data sets on which they are trained.
Keywords: Reprodueibility, Replication, Curse of Dimensionality, Intrinsic Dimension
1 Introduction
Machine lcarning (ML) is a rapidly evolving field that has made significant contributions to numerous industries. In view of its considerable impact, also becomes apparent how difficult it is to replicate and reproduce empirical findings the field of ML. Therefore, reproducibility in its own right. Reproducibility, defined as the ability of researcher duplicate the results of a prior study using the same materials as the original investigator, is critical to ensuring the validity and reliability of research findings. It promotes transparency, allows for verification of results, and fosters confidence in the scientific community. Despite its importance, achieving reproducibility in ML rescarch is challenging due to several barriers. One of the main difficulties is the implementation of the exact experimental and
ML is an important topic
"Corresponding author

---

Published in Transactions on Machine Learning Research (10/2024)
computational procedures described in the original work. The resulting layers of complexity become particularly apparent when the used computational frameworks continually update and rise and fall in popularity and levels of maintenance. Another major challenge is the inherent instability of results. This is influenced by a multitude of factors such as the amount of data available, the computational resources at hand, the determination of hyperparameterss and the inherent randomness of the training process. In this context, even more difficult. to assess the influence of uncontrolled epistemic uncertainties, such as the intrinsie dimensionality. Several guidelines, originating from conferences, workshops, and coding frameworks, provide recommendations and tools that help researchers and authors in this regard (Pineau, 2020; ICLR, 2019; Lightning AT and Contributors, 2022). However, these are often not very detailed or only allow a limited structural evaluation and comparability of reproducibility. Moreover, several authors note a lack of standard Lerminology for reproducibility (within ML), which hinders the emergence of an unified evaluation framework (Tatman et al., 2018; Bouthillier et al., 2019).
This paper proposes a comprehensive and in-depth framework for the study of reproducibility in the rescarch area of graph neural networks. The challenges associated with dala sel, melhod and resull are analysed in terms of their significance for computational reproducibility. A multi-stage selection process identified six scientific papers for which we studied and adapted our framework. With their help, we explore and demon- strate the limits and difficulties of reproducibility. This results in a new ontology for scientific reproducibility that generalizes to the realm of machine learning as a whole.
A second major challenge for the reproducibility high-dimensional ML results is the occurrence of episteriic uncertainties. This is particularly the case when an attempt is made transfer a result to new data or use cases. A particular instance of this uncertainty is the umbrella term curse of dimensionallty. This is based various mathematical observations in high-dimensional spaces that are generally not addressed by ML studies. A geometric approach towards understanding the curse of dimensionality was established by V. Pestov. He proved that the concentration of measure phenomenon (Milman, 1988; 2000) contributes to the overall curse of dimensionality (Pestov, 1999; 2007b;a; 2010b;a). His approach was adapted towards a practical computable function for estimating the intrinsic dimension (ID) of a geometric data set (Hanika et al., 2022). This result was further improved with regard to its applicability to large data sets (Stubbemann et al., 2023a;b).
With regard to reproducibility, we invest igale the influence of the ID on the ML training process. In partic- ular, we experiment with ID-based feature selection, as it provides a straightforward method to manipulate the ID of a data set. As we hypothesize that training methods are susceptible to ID-changes in the underlying training data set, we apply different ML methods to the same manipulated data sets. We thereby study the impact of altering the intrinsic dimension of graph data sets for all six reproduced graph neural network methods.
Although there are studies on these theoretical and practical aspects, the present work aims to bridge the gap between them by focusing on reproducibility and the intrinsic dimension within geometric understanding. Overall, our work contributes to improving the quality and reliability of ML research, ultimately benefiting the broader scientific communily and industryy applications.
To summarize our contributions:
We introduce an ontology of reproducibility in Machine Learning (Section 3).
We consider about 100 publications from the field of graph neural networks and repro- duce six of them extensively (Section 4).
We investigate how the change of the geometric intrinsic dimension in data sets effects the performance of the six reproduced methods (Section 5).
2

---

Published in Transactions on Machine Learning Research (10/2024)
2 Related Work
Reproducibiliyy and Replicability
Several publications have investigated the general state (National Academies of Sciences, Engineering and Medicine, 2019) and challenges (Nature Special, 2018) of reproducibility and replicability in science. There is also work that has looked more specifically at these issues in the field of computer science (Freire et al., 2012) and its subfield of machine learning (Raff, 2019; Liu et al., 2020a; Chen ct al., 2022). In recent years a growing number of conferences include dedicated tracks for reproducibility efforts or specific work- shops (Stodden et al., 2013; ICLR, 2019). The knowledge collected there is now available in straightforward checklists (Pincau, 2020) and general reports (Pincau et al., 2021). Related to this more and more journals and publisher provide specific editorial policies (Casadevall & Fang, 2010; Springer Nature, 2020) to help authors in that regard. Efforts for reproducing and replicating past works from broad range of research fields concentrate in some dedicated journals (ReScience C, 2023), in which publications from further back are also of interest.' Beyond the space of academic publication there are of course similar efforts made by the programming community (paperswithcode, 2021; Sinha & Forde, 2020). Specifically, machine learning engineering teams and individuals are building frameworks (Lightning Al and Contributors, 2022) and tem- plates (ashleve and Coutributors, 2022) to strammline the process of setting up reproducible machine learning experiments. Recent research has shown that the choice of the machine learning framework used and its version (Pham et al., 2020; Shahriari et al., 2022), or the commercially available platforms providing related services (Gundersen et al., 2022b), can have a significant impact on the reproducibility properties of the research code. There are, motivated by practical concerns, surveys that. investigate directly the availability and operability of research code (Collberg et al., 2015). Few works additionally try to construct taxonomy of those reproducibility properties (Goodman et al., 2016; Kitzes et al., 2018; Tatman et al., 2018; Bouthillier et al., 2019). The accompanying discussions often emphasize the confusing terminology (Peng, 2011; Plesser, 2018; Gundersen, 2020). The provided taxonomies usually consist of a shallow hierarchy of different levels of reproducibility which are characterized by high-level features of the submissions that have to be assessed. In most cases the process of evaluating is guided by only a few questions. As such they give researchers and reviewers not that much guidance when evaluating the degree of reprodueibility. However there are publi- cations that go more into detail when analysing factors and variables that influence reproducibility (Ivie & Thain, 2018; Gundersen et al., 2018; Gundersen & Kjensmo, 2018; Gundersen et al., 2022a). This focus central aspects of computational reproduciblity can also be found in the present work.
Intrinsic Dimension and Feature Selection
The term ântrinsic dimension (ID) has multiple slightly different meanings in related sub-fields of machine learning. They share the motivating aspect of using the value of the ID of data as a proxy for gaining evidence on how the data is structured. One prominent usage of the term is for specifying the often approximated dimension of a hypothetical embedded manifold in the data space which describes almost all samples with sufficient accuracy (Hein& & Audibert, 2005; Tatti et al., 2006). This notion of ID can be used to motivate a variety of estimators, for example based sampling around data point neighborhoods (Kim et al., 2016). Those estimators give rise to different feature selection methods (Traina et al., 2010; Mo & Huang, 2012; Suryakumar et al., 2013; Golay et al., 2016), occasionally based on gradients to learn embedding with the desired properties (Pope et al., 2021). Other work has focused on the ID of the learning process or the learned model itsolf. One line of rescarch is the analysis of the ID of activations in different layers in deep neural networks (Ansuini et al., 2019; Doïmo el al., 2020). There, the notion of ID is slightly adapted to be the minimum number of coordinates required to describe set of points without significant loss of information. On the one hand, the mathematical derivations are often specifically tailored to neural networks and, on the other hand, only in few cases do they explicitly depend on the chosen feature sets employed by the model architecture. However, these algorithms do not help to decide if and to what extent the data set is affected by the curse of dimensionality and the related concentration phenomena (François et al., 2007; Houle, 2013).
1 SSee the years challenge
3

---

Published in Transactions on Machine Learning Research (10/2024)
In contrast, the intrinsic dimension for data (Pestov, 1999; 2007b; 2010b; 2011) gives an axiomatic ap- proach on quantifying the direct influence of the curse of dimensionalily by linking it fundamentally to the phenomenon of concentration of measure (Gromov & Milman, 1983; Milman, 1988; 2000). These mathe- matical works give rise to the intuitive view on the curse of dimensionality as the phenomenon of features concentrating near their means or medians, so that algorithms are therefore not able no discriminate the data. This approach by itself was computationally infeasible until the introduction of the intrinsic dimension of geometric data sets (Hanika et al., 2022) Although a geometric data set necessitates the inclusion of a set of feature functions, it affords the theorist practitioner considerable flexibility in adapting the ID to emphasize a particular aspect. Consequently, the resulting ID is highly contingent upon this initial choice. For example, when placed within a geometric data set together with geodesic distances as feature functions, the Euclidean n-sphere exhibits an intrinsic dimension in the order of n. Thus for these feature functions the obtained value is consistent with other mathematical notions of dimension. The complexity of evaluating feature functions varies, and thus some classes of functions are more usable than others. Later publications provided algorithms for practical computation or approximation of this ID (Stubbemann et al., 2023a) and its application to feature selection (Stubbemann et al., 2023b), even for large-scale data sets.
Studies Regarding Influence of Data on Model Behavior
Machine learning models are heavily influenced by the quality and nature of the input data they are trained on. This relationship has been extensively studied in various contexts. A large body of literature is concerned with influence of simple data augmentation (e.g. cropping, rotating, stretching for computer vision data sets) when keeping a machine learning model fixed (Salamon & Bello, 2016; Perez & Wang, 2017: Tsuchiya et al., 2019; Tian et al., 2020; Laptev et al., 2020). Naturally there are also works studying influence of feature selection methods (Koçak et al., 2019) and projection methods (Wan et al., 2021) in addition to those referenced in the previous sections.
In the realm of classical machine learning, theoretical studies have been conducted various models, including decision trees (Syrgkanis & Zampetakis, 2020) and quadratic classifiers (Latorre et al., 2021), that explore the estimation capabilities and the performance within high-dimensional settings. These models often exhibit dependence on dimension, particularly in the context of -igh-dimensional regimes where their effectiveness may vary. Similarly, research has been dedicated to understanding the behavior of support vector machines in spaces with low (box-counting) dimension (Hamm & Steinwart, 2020).
In a different vein, the concept of influence function is used track the impact of training data on learning algorithms, providing insight into how the model's predictions test data are influenced by the training data. When applied to neural networks, influence functions shed light on the backpropagation process and the attribution of training data importance in these complex architectures (Koh & Liang, 2017; Pruthi et al., 2020; Akyürek et al., 2022; Hammoudeh & Lowd, 2022).
Otherwise, there seems to be a lack of research on data manipulation methods that focus on the influence of the concentration of measures phenomenon on model performance.
3 An Ontology of Reproducibility in Machine Learning
The term reproducibility is often used ambiguously and vaguely in the field of machine learning (Peng, 2011: Kitzes et al., 2018; Plesser, 2018). In our work, we apply the "classical" understanding of reproducibility science. That is, whenever a scientific study is replicated the original experimental results should be achieved with high degree of reliability. Of course, the concept of "replication" implies series of attributes. Among other things, these can be the independence of the project (e.g. different group of researchers), use of different set of equipment (hardware or software), or the time since the original study was realized. As such one can see that particular scientific results can be placed on a spectrum of reprodueibility. To give a more explicit process of contextualizing reprodueibility we introduce in the following section an extensive ontology of reproducibility for the realm of machine learning. The necessity for this is basod on the fact, that, to the best of our knowledge, such an ontology does not yet exist. The components of this ontology are influenced and inspired by the Chapter Assessing Reproducibility of the online version of the book The
4

---

Published in Transactions on Machine Learning Research (10/2024)
scientific
result
theoretical
claim
empirical evidence
theoretical evidence
computational result
data set
software
availability
transformation
environment
source code
usage
model
predictions
Figure Top levels of the reproducibility ontology.
Practice of Reproducible Reseorch (Kitzes et al., 2018). Additional ascendancy comes from existing efforts characterize computational reproducibility (Gundersen al., 2018; Gundersen Kjensmo, 2018; Gundersen et al., 2022a). We adapted them with stronger formalization, giving structure to the proposed questions and adjusted the ontology to better meet the requirements for the subsequent reproducibility study. Other noteworthy influences come from an ontology for semantic terms in machine learning (Publio et al., 2018), a practical taxonomy of machine learning (Tatman ct al., 2018) which however has no formalization and very few specific points to check, and the machine learning reproducibility checklist (Pineau, 2020). One commonality between this ontology and the above-mentioned works is the focus on reproducibility of an individual research project. For simplicity, consider only the setting where computational result is presented evidence for one scientific result.
3.1 Overview
We now present our ontology reproducibility machine learning. It connects possible errors or difficulties that could arise when Irying to reproduce the results of a single scientific study. Our proposed ontology is structured hierarchy and starts on the top level with the general notion of scientific result. Such result are (subsoquently) interested in rescarch that
can be based empérical or theoretical evidence. Because uses experiments for producing evidence we do not subdivide the theoretical category. In contrast, propose fine-grained structure for the empirical category. To reflect the related central aspects of data processing we use the ontological entities data set, software and computational results as the main subcategories for the empirical category. Each of these aggregate again a set of subentities. For example, the data set category encompasses the availability and transformation of data. Similarily, the software category has as rentral subcategory source code but also includes environment and usage as subcatcgorics. Most importantly, we include the derived model and its predictions in the computational result calegory. Figure shows the schema that provides overview of the main categories and their subcategories.
We further evaluate every considered research paper within the proposed ontology based on set of questions. In the following we want to describe the formalization of these questions and their molivations, which are based on potentially occurring errors and their impact on the scientific reprodueibility. As we follow an open world semantic with our ontology the questions are formulated in such way that answering them negatively is good for reproducibility. This also means that in the subsequent questioning, a missing answer does not indicate the non-existence of the corresponding property. We have included some questions that
5

---

Published in Transactions on Machine Learning Research (10/2024)
remain unanswered for all the publications surveyed (a good thing), but could be helpful in obtaining a finer distinction of reproducibility when the situot they cover occurs.
3.2
Data Set
One central aspect of scientific result machine learning the data set to which previous and proposed methods are applied. A common approach structured data sets. Reprodueibility is only possible with detailed knowledge about. the process for obtaining and preparing used data sets.? 2 Explanations and antomatization minimize the risk of working with data set that follows different distribution than the one used in the original study. Within this category of the ontology, we want to focus on whether the publication and accompanying material include steps (manual or automated) that describe how to obtain data sets.
evaluate methods over multiple different but similarly
3.2.1 Availability
The questions from this category aim reflect nuances in obtaining and understanding the data sets used by the publication.
D1 Is the data set format not documented? Applicable mainly for publications that introduce a new or heavily modify an existing data set. Otherwise it is difficult to adapt methods for re- producibility.
D2 Was the data set version not set explicitly? Over time data sets might undergo changes. Indi- vidual samples can be relabeled, new samples included or others removed through further cleanup. These modifications lead to the need to keep track of the used version of the data set.
D3 Was the data set not directly accessible? Since most the data sets currently used in machine learning are available on the web, direct download link good start for reprodueibility.
D4 Did the access not work at time of study? Unfortunately, the hyperlinks provided no longer point to the original resource sometimes.
D5-Does the data set have privacy restrictions? Although not very common in widely used ma- chine learning data sets, privacy concerns may lead to access restrictions.
D6 Does the data set require restrictive license agreement for accessing?
D7-Is the data set available on request only? This is loosely linked to the previous point. Some- times it might be necessary to go through a more elaborate process to obtain the data set. Such steps are often quite brittle over time and are unlikely to be maintained in the long term
3.2.2 Transformation
In general data set needs to be adapted before a method can be applied. The questions from this category deal with evaluating those pre- pre-processing steps.
D8
Are manual steps necessary for pre-processing? A series manual steps to transform a data set could easily be a source of error, an automated solution is preferred.
D9-Is there only an incomplete description for pro-proccssing steps? The provided explana- lions or scripts might not be enough to gel the data set in the necessary form. This point is especially crucial for lesser known data sets. For such data sets, it is particularly helpful for further investigation if tools are provided for loading and accessing individual samples.
But even then there special cases where methods select data samples generates them (e.g. active learning, reinforce- learning), and reproducibility of similar aspects handicapped by other means.

---

Published in Transactions on Machine Learning Research (10/2024)
P10-Are the train, validation and test splits not clearly defined? Usually only part of the data set is used for training, whereas other parts are used for validation and testing. The process of allocation should be reproducible, be it through provided files or deterministic functions.
D11-1 Is the number of samples not documented? An easy way of checking one attribute of the transformation is counting the obtained samples, Additionally it gives a high-level overview over the data efficiency the presented approach.
3.3 Software
The implementation and application a central part of a proposed machine learning method. one aspect of the research protocol and acts as description of the executed experiments. The software code operates on one more data sets and produces computational results. In this category combine aspects of the code written by the authors, ancillary software, and other components crucial for reproducibility. This keeps the ontology clearly structured.
3.3.1 Environment
Questions from this category deal with general behavior of the target system, which heavily influences the context of exccution of the experiments.
S1 Is the exact version of dependencies not documented? Multiple dependencies can interact in intricate ways. This makes pinning of exact versions necessary for avoiding possible bugs connected lo incompatible versions as well as prevent time consuming fixing of conflicts.
$2 Is the specified version of dependencies not available anymore? Depending on the age of the publication and the lype dependency used, old versions may have disappeared from the distribu- tion channels. For smaller projects, they may no longer be maintained by the project's developers maintainers.
$3 Is necessary hardware unavailable? Many specialised hardware requirements can be circumvented by simulating a computational environment with virtual machine container images. However, this adds a time-consuming overhead when trying to achieve reprodueibility, is not feasible in a reasonable time.
S4 Are any seeds for random number generators not set? Multiple dependencies each have different random number generator, where each has to be set for getting closer to reproducibility of an experiment.
S5 Are important variables unclear? Some settings of an experiment run (e.g. number of GPUs used) can have significant impact on results or even side effects onto other settings.
3.3.2 Usage
This category of the ontology groups together aspects regarding how the experiments were started. For the set of considered machine learning papers it is not necessary to consider user input beyond starting configuration.
S6 Is the documentation not up-to-date? Few publications include dedicated documentation of the source code they provide. If only a simple Readme file includod, it should at least not be mislcading for the reproducibility attempt.
S7-Are necessary arguments not clear? Depending on the implementation, some arguments may be required to run an experiment, but neither the defaults nor the values used are explained or provided.
S8 Are there missing hyperparameters? Similar to the previous question, the values of the hyperpa- rameters for the experiments are generally important when reproducing them, they significantly influence the result.

---

Published in Transactions on Machine Learning Research (10/2024)
S9 Are train/test scripts incomplete? Inchuding the individual commands of experiment in sin- gle file usually makes it easier to attempt reproducibility. How to start these scripts can be source of uncertainty if certain flags or variable values used in the provided script are missing, incorrect, corrected later and/or not explained at all. In addition, if pre-processing steps are not inchuded or other sleps the computational pipeline are missing, reproducibility is compromised. It may also be the case that not all experiments presented in the paper have scripts that run them.
S10
Is it unclear which version of scripts was used? It is only natural for the main source code of the release to be changed before (and after) publication accommodale for bugs and reviews. Problems arise when these changes not reflected in the accompanying scripts or instructions, when multiple scripts exist for the same experiment.
3.3.3 Source Code
The questions in this category concerned with the source code files that implement the ideas of the scientific experiment. We do not have separale questions regarding the availabilily of source code itself similar to the data set category because we only focus on papers that provide source code with non- restrictive license.
S11-Is there a bug that was never fixed? Over time, authors and external contributors may find dif- ferences or errors belween the original publication, its revisions and the implementation.
$12 Are there issue solutions that were not applied? Usually, the discovery of implementation problems is accompanied by public discussions the website hosting the implementation. However, it may happen that the solution discussed has not been implemented, or has not been merged from an external source fork.
S13 Was a bug fix distributed through other channels? On the other hand, public discussions may indicate that the fix was distributed (manually) through some other channel, such as emails or direct messages
celected/active group of participants. In these cases, is not clear what the detailed changes are, where they are, and how they affect reproducibility.
S14 Did the API change? This question relates to attribute $10. We now considering the converse sccnario, wherein the entry points to parts of the implementations have undergone alteration, yet this is not rellected in other pertinent sources, such documentation supplementary materials. Furthermore, the reproducibility of results is hindered when the traceability of versions is constrained by a complex and unclear history within the version control system.
515-Did an out of memory error occur? As a special type of defect, it is only recorded when there are no or incorrect requirements. As our aim is to assess reproducibility in general, we do not determine the specifics that caused this error.
S16 Are steps for one experiment missing? the necessary source code for an experiment is not included, the reprodueibility of this experiment is more difficult to achieve.
S17 -Are parts of the source code not available? In addition to the previous point, want to eval- uate the possible situation where the publication uses libraries or code that is not included in the source code provided. This makes reproducibility almost impossible.
S18 Is the hyperparameter search not included? As hyperparameter search is an integral part of experiments, it is important for reproducibility to have an explanation or process in the implemen- tation of how the search was performed.
S19 only the general idea (and no experiments) implemented? Another reason for the lack of experiments could be that the publication simply proposes new machine learning algorithm or a building block for an existing one.
8

---

Published in Transactions on Machine Learning Research (10/2024)
3.4 Computational Result
The result obtained from computational experiment is the evidence in support of a scientific claim. The full reproduction of a scientific result and its evaluation depends on the successful completion of the repro- ducibility steps for data set and software. If this is not possible, no model or predictions can be obtained for further evaluation steps. Conversely, if the supplementary material of the publication in question contains the learned model, it still possible to perform the subsequent reproducibility steps. However, this is a rare case. Ideally, current scientific results in machine learning should be reprodueible in terms of data set and software, well as providing the learned model. This will allow for more in-depth comparison and analysis wetween the reproduced and the provided models.
3.4.1 Model
As outlined above, casy access to model weights necessary for full reproducibility, espocially when other factors increase the difficulty of obtaining optimized model independently Furthermore, the practical problem of making the model available still has no ready-made solution. There are few existing platforms that allow the combined hosting of source code, data sets and models. Limitations can quickly become apparent when data sets and models are large, or when multiple of them are used or shared. For now, there is only one question in this category.
R1 Are there no parameters (weights) of the obtained model provided? As much of today's machine learning approaches use larger data sets and models, it takes more and more time and computational resources to run the proposed method. Making model weights accessible can there- fore act as a kind of shortcut if the focus is on comparison with other approaches. Depending on the programming language and language and format, gives other researchers an straightforward way check model specifications. addition, when reproducibility fails, provides a way to find the cause and, more importantly, to least verify the author's claims.
3.4.2 Predictions
This category is intended to capture aspects of the outputs of the model when evaluating over the train or test data, and how these affect reproducibility. Depending the results presented, a comparison of different evaluation metrics may be helpful, especially if the inference requires more time or resources than are available for the specific reproducibility atlempt. We note that all but the last question are inherently hard to quantify.
R2-Are there small deviation to obtained model? We will measure the differences by comparing the central evaluation metrics between values reported by original authors with evaluated metrics on the reproduced model. We answer the question positively, when the relative difference in the range of +1-2%.
R3-Are strong differences in few experiments observed? Similarly to the previous question we as- sign this attribute when the difference of evaluation metrics is in the range of +2-20%
R4 Are strong differences in almost all experiments observed? As an extension of the previous question, assign this attribute when almost reasonable reasonable reproducibility of the results.
R5 Are the claimed results only supported by small sample size? Individual runs of a machine learning algorithm are rarely exactly reproducible, even with the best offorts to achieve repro- ducibility, both by the original authors and those who reproduce the work. Therefore, averaging over several runs with the same configuration of experiment will strengthen the reliability of the result. We mark this attribute if there are less than five such similar runs per result.
R6-Are there no predictions (outputs of classes or decisions) on the data sets? If the original publication provides the class predictions or decisions made by the model, the reproducibility attempt provides the opportunity to examine more comprehensive metrics for differences in model behavior.

---

Published in Transactions on Machine Learning Research (10/2024)
Although it is not always feasible to provide the full set of predictions (e.g. for large data sets methods from other fields such as reinforcement learning), it could be done for non-trivial parts of the data set.
3.5 Limitations and Extensions
It is apparent that this ontology is designed using basic formalization language. All connections between the entilies can be read as part of. The authors are well aware of the ML-Schema (Publio et al., 2018). However, we decided not to include or build on it, because: (i) several aspects of reproducibility could not be expressed using the ML-Schema; (ii) the focus of the ML-Schema is on sharing information about machine learning algorithms, and not on the reproducibility of a scientific result. An example of this is the lack of consideration of the influence of the seed for the random number generator.
There are a number of possible modifications or extensions that we have not included in the present version of the ontology. For example, our ontology does not take into account detailed information about theoretical evidence. This is mainly motivated by the survey in Section 4, which focuses on empirical evidence. In the category of empirical evidence, the name software is somewhat misleading, as it also includes aspects related to hardware. One could rename this category or add hordware as a separale sub-calegory of empirical evi- dence. Similarly, the entity source code could be extended to include details of different modes of availability and documentation. special attribute might be the use of version control software.
Certain aspects of reproducibility are not yet considered, e.g. that plots, figures and tables can be generated automatically. Authors often fail to provide the source code for the visualization. Furthermore, our ontology does not capture aspects of data provenance.
In addition, it seems that the longer the time since publication, the lower the achievable level of reprodueibil- ity. This may be due to the ageing of the hardware and software used in the experiment, which may no longer be available.
Finally, in our ontology we treat the presence absence of an attribute categorically. Therefore, in cer- tain cases, evaluating research paper using our ontology is difficult task. Conversely, any subsequent ontological operation and explanation is independent of any interpretation of numerical values.
Furthermore, it is notable that a reprodueibility score for a publication, which could be derived from the evaluated attribute, is absent. By maintaining the evaluation through the ontology in a qualitative manner thus far, an ordinal measuring structure has been obtained that allows for formal comparison and ordering. Additionally, the ordinal evaluation framework allows for the explanation of differences in reproducibility or lack thereof. Neverthcless, it would be advantagcous to have scoring function in place. However, at this moment, we are unable to define scoring function that we can guarantee will be meaningful.
4 Reproducibility of Major Graph Neural Network Research Results
The first. main goal of the present work to achieve a scientific overview over the state of reproducibility in the research field of graph neural networks. For this we first depict our method of candidate selection in Subsection 4.1 and thereafter discuss all our findings with respect to our reproducibility ontology.
4.1 Candidate Selection
The main criterion for selecting paper was its impact on the research field of graph neural networks. In the following we describe in detail our procedural steps. As lower bound for the publication year we selected 2016, the year of the publication of the seminal GCN paper (Kipf & Welling, 2016). On the other hand we considered works that were published before 2023. Most importantly we required that the paper in question has an experimental evaluation, due to the overall objective of the study to investigate the influence of intrinsic dimension. We also included research works that were only in the preprint stage.
As the citation count is an often used proxy for measuring scientific impact, and the same time readily available, we employ it in our selection process. In detail, we the Semantic Scholar (Allen Institute for
10

---

Published in Transactions on Machine Learning Research (10/2024)
Artificial Intelligence, 2022) search engine for selecting papers based their average citation count since their publication. 3 In our selection process we discarded all papers without publicly available source code for their experiments. We further discarded papers that either covered implementation details of software librarios or focused on applications existing methods. Finally we refined our selection with the help of several domain experts that pointed us to important research works from the domain of graph neural networks.
Our selection process started out with 9223 unique candidates. We then calculated each paper's score and selected the 100 papers with the highest scores. We manually ignored works that did not propose new method in the field of machine learning. This included surveys, coding frameworks, and works that only applied Graph Convolutional Network (GCN) methods to other field of science. Additionally, publications applying methods to very specific data sets and those with time-dependent or spatial data were not included.
Out of those remaining 55 results we applied the source code criterion and arrived at 42 papers. In Figure 2 we depict the yearly distribution of the number of papers (a) and their score distribution (b).
Now the following limitation of the selection method becomes more apparent. As citations are distributed over publications and there is an increasing number of papers published each year, older papers have advan- tage over newer ones. Conversely, the evaluation function dampens the infiuence of older publications to a much lesser extent.
On the other hand we wanted our selection method to reflect a "normal" search behavior of a researcher. The power-law distribution of citations is well known property of citation networks (Price, 1965) and also somewhat expected because they social networks which accompany the scientific process. More specifi- cally methods of more frequently cited papers are chosen more often than those with less citations (Hazoglu et al., 2017). The power-law distribution of the citations (and subsequently the score) motivated selecting only a few papers as those publications had the majority of the impact on the field measured by the above method.
2000
Selected Papers
3-
Score
Years
Papers
(a) Number of selected papers per year.
(b) Selected papers sorted by score.
Figure Visualization to get an overview of the distribution of the selected papers.
We provide list of considered papers in Appendix A in the appendix. We started with selecting candidates for the reproducibility survey before deciding the automated process presented above.
The primary factors considered in assessing the reprodueibility of a given paper at this stage were incompat- ibility of hardware requirements and errors in the source code provided. these issues could not be resolved with reasonable effort, even with experience using the libraries, the paper was deemed not reproducible. Unfortunatcly, we have not been able fully convert the previous manual pre-sclection into an process that
In other words, rank search results for keyword graph neural network hased the score: 2023 year publication
number citations
11

---

Published in Transactions on Machine Learning Research (10/2024)
is itself reproducible. The difficulty in reproducing the results due to the uncontrollable non-determinism introduced by the usage of the Semantic. Scholar A AP over multiple runs. Furthermore new citation data changes in the rankings over time.
Making things worse, the obtained collection did not always include all papers that we tried to reproduce. However, once we got a set of considered papers, we refrained from further optimizing the selection process to include all reproduced or not reproduced papers. We acknowledge the possibility of producing a positivity bias by excluding publications where reproducibility failed completely. However, we believe this is preferable to conveying skewed perspective of the reproducibility of the survey itself and the field of graph neural networks in general.
Therefore, the collection contains 6 publications, for which we completed the complete assigmment of the described reproducibility attributes. SGC and GraphSAGE were successfully reproduced candidates deter- mined from prior iterations of the sclection process. Since we already had experience with the publication for SAGN+SLE, we included it well. The final selection of reproduced papers can be seen in Subsection 4.1 together with their abbreviations used in the following.
Table Reproduced Papers and the abbreviations used.
Abbreviation
Title
Reference Key
GCN
Semi-Supervised Classification with Graph Convolutional
Kipf & Welling (2016)
Networks
R-GCN
Modeling relational data with graph convolutional networks
Schlichtkrull et al. (2018)
GraphSAGE
Inductive Representation Learning on Large Graphs
Hamilton et al. (2017)
DillPool
Hierarchical Graph Representation Learning with Differen- tiable Pooling
Ying et al. (2018)
SGC
Simplifying Graph Convolutional Networks
Wu et al. (2018a)
SAGN+SLE
Scalable and Adaptive Graph Neural Networks with Self- training
Sun & Wu (2021)
4.2 General Observations with Respect to our Ontology
Reproducing experimental results from the selected scientific research was challenging due to various factors. It is usually the case that the papers alone do not provide enough information to replicate the experiments independently. Accordingly, we have always started the reproducibility attempt with the associated source code. One major issue is that those repositories often lack crucial dependency information (S1), making it difficult to even run the entry point scripts without errors. To overcome these challenges, it was necessary scarch through the accompanying discussions and seek clarification that might be missing not working correctly. Additionally, it is often the case that the commands provided in the simple documentations for running experiments rarely work as expected (S6, S7). Furthermore, the availability of different data sets adds complexity to the reproducibility process, especially considering that many publications were published before coordinated efforts to unify the data set landscape, for example the Open Graph Benchmark (Hu et al., 2020a). However, data sets are generally accessible, although is rarely the case that the preprocessing steps are explained (D9). Another common point is the aspect of hyperparameter search, which is not included in most of the software provided, even if it is mentioned in the publication (S18). Finally, it is common for papers to fail to provide the model (RI) or the predictions of the model on specific data (R6). The following is a more detailed description of specific problems, grouped under the main categories of data set, software and computational result, encountered during the reproducibility attempts. For a better overview, we will focus on selected points that stand out.
the exact parameters and commands
We would like to emphasize that in our ontology is not advantagcous have an attribute, as this is evidence that it is more difficult to reproduce. In cases where it was not clear whether an attribute was present, we chose not to disclose it.
12

---

Published in Transactions on Machine Learning Research (10/2024)
4.2.1 Category: Data Set
We observe that the data set used in the rescarch is conveniently available it is included in the repository. However, there is lack of explanation regarding the preprocessing steps (D9). Despite this, working function is provided, which can be used for the transformation process.
GCN:
R-GCN: A similar point in GCN regarding (D9) applies.
GraphSAGE: The availability of the web of science data, set is limited to those with the corresponding license (D6) and upon request (D7). Additionally, manual preparation (download) of the data sets is required (D8).
DillPool: The implementation ation does not use proper train/ /les splits because the approach is only evaluated with k-fold validation (D10).
SGC: A similar point in GCN regarding (D9) applies.
SSGN+SLE: Except the general observations of missing explanation of the preprocessing steps (D9) the aspects of the data category are sufficiently reproducible.
Table 2: Observations for data set category with respect to 4.2.1.
data
avatiability
traneformation
metadata
downlond
selection
D1
D2
D3
D4
D5
D36
177
138
D9
D10
D11
GCN Relationa) GCN
Graphãage
Diffpool
SGC
SAGN+9LE
4.2.2
Category: Software
The dependencies are not properly specified (S1), which made it challenging to set and run the experiments. It is worth noting that the documentation is not up to date (S6) and contains mislcading information.
GCN:
Firstly, there is no requirements file provided (S1), making it challenging to recreate the neces- sary environment. Additionally, information about the specific Python interpreter version used is hidden. Furthermore, the seeds for randomization are not set (S4).
R-GCN:
GraphSAGE: The necessary arguments for the evaluation scripts not stated (S7), leaving researchers unsure of the required inputs. Furthermore, there are discussions about possible values, adding ambiguity to the code (S8). Additionally, the evaluation scripts themselves are incomplete or misleading, further hindering reproducibility (S9). The software used in the study has some bugs that affect reproducibility (S11). For example, the evaluation script for the ppi data set is incomplete, but a fix is available pull requests (S12).
DillPool: Unfortunately there is no explicit list of necessary requirements (S1) and only a minimal README file (S6). Additionally it seems that the provided commands do not work (S7, S8 and S9) and that the seeds for randomization are not set before the experiments (S4). The implementation also did not include steps to reproduce two experiments with the reddil- 12k or collab data sets (S16).
SGC: No features of that category that hindered the reproducibility were observed.
SAGN+SLE: Again, there is no requirements file provided (S1). Unfortunately we encountered out-of- memory errors (S15) when trying to reproduce experiments using data sets ogb-papers and ogb-mag. Tt could
13

---

Published in Transactions on Machine Learning Research (10/2024)
be argued that this would mean that the necessary hardware is unavailable (S3) but maybe it could be fixed by changing hyperparameters like batch size.
Table 3: Observations for software category with respect to 4.2.2.
sollware
environment
nource
dependancies
variables
documentation
scripts
experimeenta
82
83
85
80
87 | se
89 (110
| 813
812
813
814
815
8116
817
S18
S18
GON Relational GCN
CraphSage
X
Dillpool
SGG
SAGN+SLE
4.2.3 Category:
Computational Result
When examining the results, it observed that there are small deviations the Lest set accuracy, typically within a range of +1% (R2). However, it appears that only single run was conducted for each experiment,
GCN:
standard deviation is provided for the final performance. Although the authors claim to have run the experiments with multiple seeds, there is evidence of this in the code, which raises concerns about the robust of the reported results (R5). We were not able to reproduce the experiments with the neil data set because of the difficulties to prepare and use the data set.
R-GCN: A similar observation regarding multiple runs was made for this publication well. Additionally, the results of the study exhibit both small deviations and strong differences in different parts. For the AIFB, MUTAG, and AM data sets, small deviations of approximately +2% in accuracy are observed (R2). However, for the BGS data set, significant difference of 15% observed, indicating a substantial variation in the results (R4).
The results of the study exhibit small deviations, typically within range of =2%, for the available data sets (R2). However, the code indicates that the experiments were only run once (R5).
GraphSAGE:
DifiPool: Due time constraints we were only able to reproduce the experiments for the DD and Enzymes data set and observed small deviations 2% (R2). Even though k-fold validation used the experiment only run once (R5).
SGC: No feature other than the usual were observed.
SAGN+SLE: Similar to other survey candidates we obtain results that exhibit small deviations, typically within a range of +22, for the available data sets (R2).
Table 4: Observations for compulational result category with respect to 4.2.3.
computational reoult
predicsions
R2
R5
GCN
Relational GCN
GropliSage Diffpool SGC
X.
8AGN+9LE
14

---

Published in Transactions on Machine Learning Research (10/2024)
4.3 Discussion
Regarding the reproducibility ontology, observed that most paper look almost the same in the data set category, but required quite different efforts in the reproducibility attempt. This is particularly visible in the fact that attributes D1 to D5 are not present in any of the surved articles. The ease of reproducibility was mainly decided by the information provided in the README document of the source code. This means that the ontology does not capture this aspect very well for this category, even if there corresponding attribute in the source code category.
The software category, the other hand, allowed for very good differentiation of the different papers in terms of their reproducibility. Again, some attributes do not seem to contribute to the decision on the degree of reproducibility. However, further attempts to reproduce other publications may find these currently unused tribuuts useful.
The computational result category has some attributes that are common to all reproduced papers. This suggests that the corresponding properties (providing model weights and predictions) are the most difficult to obtain. We observed several papers where the number of repetition for the experiment in the original paper was low. This could be due to higher computational roquirements. We also found that we had included too few cases of possible evaluation scenarios, and that those that were included were too broadly defined. Furthermore, there are no attributes to assess the degree of reproducibility of follow-up or downstream tasks, which are also addressed in the original papers.
Several of the altributes analysed reflect. the technical barriers to reproducing the work. As it rare for a single script to start all the experiments, the necessary commands have be collected manually. Finding the right combination of Python environments and libraries is also difficult due to the lack of specified versions and dependencies. Some methods require significant computing resources and time, so that some errors are only detected after a long time. Sometimes the logs generated are only available in the console, making debugging and monitoring difficult. In addition, checkpoints are not specific to hyperparameters, leading to potential overwriting during training sessions. This lack of specificity increases the effort required to reproduce graph neural network research and compromises the reliability of experiments.
5 Influence of Intrinsic Dimensionality on Model Performance
The second main goal of the present work is to investigate the influence of intrinsic dimensionality on model behavior. We begin with the mathematical foundations of the concept of geometric ID in Subscction 5.1 and then present our experiments and results.
As already mentioned the geometric intrinsic dimension (Hanika al., 2022) is a computational accessible approach for measuring how given data set is affected by the phenomenon of concentration of measure (Gro- mov & Milman, 1983; Milman, 1988; 2000), which itself is deeply connected to the curse of dimensional- ity (Pestov, 1999; 2007b;a; 2010b;a). Of central importance are feature functions that concentrate. This means that they map most of the values of their domain close to the mean or median of their image set. Pestov has postulated that features of this type contribute the most to the curse of dimensionality. In his approach, all 1-Lipschitz functions are considered as potential feature functions. Tn the revised axiomatic system introduced by Hanika et al. (2022) the notion of dimension function emerged. Such a dimension function allows for estimating the extent to which the provided features concentrate on the data set without the necessity of evaluating all possible feature functions. This is especially important in machine learning, where algorithms and, in particular, models, usually only have access to a limited range of features of a given type.
In recent works, the computation and approximation of the dimension function have been improved when choosing the set of all component projections as feature functions for data sets in Euclidean space (Stubbe- mann et al., 2023a). The objective of this study is to build upon the findings of previous research that employed the geometric intrinsic dimension for feature selection (Stubbemann et al., 2023b).
15

---

Published in Transactions on Machine Learning Research (10/2024)
5.1 Foundations of the Concentration- Intrinsic Dimension
We commence by providing brief overview of the mathematical delinitions that the intrinsic dimension builds upon. Readers who wish to gain more detailed understanding of this topic are directed to the cited works, which offer in-depth explanations. Since this work is primarily concerned with practical (and thus finite) setting, the following definition, which is specific to that context, will suffice for our purposes.
Definition 1 (Adapted from (Hanika et al., 2022)). Let D = (X, μ) be a triple consisting a finite set X of data points and a finite set F T of feature functions from X R. Consider the function de y) supfer [f(x) - f(y)l- We require that supz.yeX dr(z,y) oo is fulfilled and (X, dr) is a complete and separable pseudo metric space with y being the normalized counting measure on (X, dr). We call D (finite) geometric data set.
We will now introduce the building blocks that give rise to a dimension function that fulfills the aforemen- tioned axioms postulated in Hanika et al. (2022). A function of this nature will indicate a geometric data set by a low value, precisely when the contained data points can be more ellectively discriminated by the corresponding set of feature functions.
Given a feature E F we want to evaluate how can discriminate sets of a specific mcasure (o.g. size Ca = [1-x[(1-a]] for fraction e 10 of the whole X. For this we use can use the following function:
PartialDiameter (J, a)p
min max 1f(x) f(0)l-
MCX x.E c
By considering all feature from the feature set we arrive at the
ObserrableDiameter((.. -a) = sup PartialDiameter((. a.)D fEF
When considering all possible values for a we obtain a way to describe the ability of feature set F of a geometric data set discriminate data points in X:
A(D) = L. ObserrableDiamete(((.. (a) da
It turns out that we need one more step to get the dimension function we are looking for:
O(D) = A(D)2
For the case of finite geometric data sets it follows that the ID can be explicitly calculated with the help of the following expression
X
A(D)
max min max 1f(x)= f(y))-
X
JEP MEX J.E
|
Using the notation (D) = minMcx,|M w....... NEM 1f(x) f(y)), and dx((D) = maxfer ōk,s, this be rewritten as
X
X
A(D)
max ( JEP
Σ
(D))
X
5.1.1 Approximation of Intrinsic Dimension
The straightfrrwadd computation of the equations in the previous section is hindered by the task to iterate through all subsets M S X of size k. This yields an exponential complexity with respect to |X] for computing A(D). As suggested by Hanika et al. (2022) and later proven by Stubbemann et al. (2023a), we can instead use algorithms with a quadratic runtime complexity in |X] compute the ID. Furthermore, for settings where a quadratic runtime still not sufficient, the authors propose the following concept.
16

---

Published in Transactions on Machine Learning Research (10/2024)
Let s= (2 $1..... 81 [X]) be strictly increasing and finite sequence of natural numbers. We call support sequence of 'D We additionally define
A(D),, =
1
(D)
+
Σ
E
i= 415)
1-1
A(D),, = 1
Σ (D) + Σ
Σ
25 (D)
X
and call accordingly 0(D),,- == A(D),,- the lower intrinsic dimension of D and 0(D)s,t = A(D)..- the
upper intrinsic dimension of D.
This results in giving us lower and upper bounds for A(D) and thus for the ID. By using upper and lower bounds, we can obtain the following approximation of the ID:
@(D) O(D), O(D),,t + +(()).-
2
Stubbemann et al. (2023a) provides an algorithm for calculating this approximation.
5.2 Dimension based Feature Selection
The intrinsic dimension of a data set refers to a measure of concentration that captures the underlying structure or information of the data. It is challenging to quantify the impact of intrinsic dimensionality on particular machine learning method. This motivates the need to investigate its effect. One approach to achieving this is by discarding the features that have the most significant influence the dimensionality of the data set. The removal of these features allows for the observation of any changes in the performance of the trained model. This approach allows us to examine the relationship between intrinsic dimensionality and model performance. Feature selection can be regarded as a means to an end in this research. serves as tool to identify and eliminate those features that contribute the most to the dimensionality of the data set. order to calculate the influence of dimensionality and perform feature selection, we rely on methods demonstrated in Stubbemann et al. (2023b), which we will briefly include in the following paragraphs.
The discriminability of 'D with respect to feature f E F is defined as
X
A(D); = X k=
1
Σ p
Noto, that one data point with an outstanding value f(x) can have a strong influence on A(D); via drastically increasing (D). To weaken this phenomenon, we weight (D. higher for smaller values of The normalized discriminability of D with respect to which define as
[X|
A(D),
Σ
(D)
The normalized intrinsic dimensionality of with respect to f is then given by
a(D), A(D)
The higher this value is for a given feature, the more it contributes to the intrinsic dimension and thus reduces the ability to discriminate between data points.
Stubbemann et al. (2023b) provides an algorithm for calculating the normalized intrinsic dimensionality directly.
17

---

Published in Transactions on Machine Learning Research (10/2024)
5.2.1 Approximation of Discriminability
Unfortumately, before, an explicit calculation of the normalized discriminability for larger data is not feasible because the algorithm scales quadratically with the number of data points. We can, however, use similar approach to the previously referenced method of approximating the intrinsic dimension with the help of support sequences to approximate the discriminability well.
For a feature f E F and support sequence we call
1 A(D) = [X|
Σ
S (D)
Σ
Σ S
8
the upper normalized discriminability with respect to f and and
A(D),,, = [X]
Σ f=1
D (D) + Σ
Σ E
1 (D)
the lower normalized discriminability with respect to and 8.
We define the upper/Rower normalized intrinsic dimensionality with respect to f and via O(D) s.f (A(D),.)) and 0(D),,} == (A(D)t,)) Equipped with these we then can assign each feature their approzimated normalized intrinsic dimensionality with respect to f and
O(D), = ODD),, = O(D) 2, +
2
Stubbemann et al. (2023b) provides an algorithm for calculating this approximation of the normalized intrinsic dimensionality.
5.3 Experimental Execution and Impact on Intrinsic Dimension
In order to demonstrate the impact of the intrinsic dimensionality of the different data sets on the methods employed in the reproduced papers, we have chosen to discard those features with the highest (approximate) normalized intrinsic dimensionality.
In the context of contemporary machine learning, data sets typically represented by matrices. For graph data, this typically refers to the data pertaining to the nodes X. In addition, the connectivity information, represented by the adjacency matrix A, and any edge features, are also considered. However, aggregating this information into a feature matrix through the use of neighbourhood aggregation, in the form of A (where is a small positive integer), does not alter the qualitative insights provided by the intrinsic dimension, as demonstrated by previous research (Stubbemann al., 2023b). Aggregation common feature of graph neural network mothods, which often employ indirect forms of aggregation through the use of specific models. Given this, we have chosen lo refrain from considering the neighbourhoods, and instead focus on the matrix of node features of shape X d, where indicates the number of samples and d the number of attributes per sample. For each data set in our investigation we use the following representation set as introduced in Definition 1. The set X is comprised of the samples &; where each sample consists of the attributes x; = functions F. Together with the counting measure v(A) = |A//n for subset C X we complete our special instance of the geometric data D.
a goometric data
Tia). We chose the set of component selectors fs( = It as the set of feature
The sizes of all used node feature matrices can be seen in Table 5.
18

---

Published in Transactions on Machine Learning Research (10/2024)
Table 5: Sizes for all data sets and the research works they appear in, in the scope of this work.
Data Set Name
Nodes
Edges
Features
Paper Names
citesoer
3312
4732
3703
GCN, SGC
2708
5429
1433
GON, SGC, SAGN+SLE
pubmed
19717
44338
500
GON, SGC
aifb
8285
29043
4
H-GCN
mulag
23644
74227
2
R-GCN
lags
333845
916199
2
R-GCN
1666764
5988321
11
R-GCN
enzymes
19474
37282
18
DiffPool
ppi
14755
225270
50
SACN
ppi (large)
56944
818716
50
GraphSAGE, SAGN+SLE
roddit
232965
11606919
602
GraphSAGE, SGC, SAGN SLE
flickr
89250
899756
500
SAGN
yelp
716847
6977410
SAGN+SLE.
Data set preparation For cach rescarch paper, we initially extract the essential components for loading and preprocessing the data sets from the source code supplied by the authors. These components are then used to obtain the node feature matrices of the data sets used. In instances where is unavoidable, we resort concatenating the node feature matrices from both the training and test data. It is of paramount importance to note that we rigorously ensure that the machine learning method does not have greater access to the test data than was originally permitted in the original implementation.
The rationale for applying fre selection after preprocessing is as follows: Our approach aims to investigate how methods are influenced by the data on which they are applied, for example, how the model "sees" the data. Some forms of preprocessing alter the empirical data distribution, and preprocessing typically lacks any Iearnable parameters. Furthermore, the model in question rarcly has explicit information about the applicd preprocessing steps. Consequently, we do not consider the preprocessing steps be part of the model. This approach facilitates the separation of the influence, otherwise the change in the preprocessing resulting from feature selection would be included in the observations and discussions.
Feature selection We used the algorithm for direct calculation of the discriminability (Stubbemann et al. (2023b), Algorithm 1) for data sets with less than 10y samples. For larger data sets, we employed the approximating version (Stubbemann et al. (2023b), Algorithm 2). In those cases we first choose a geometric sequence = (s)....st) of length = 10, 000 with 81 = [X] and 81 = 2 and use the support sequence (Subsection 5.2.1) 8 which results from s' = (LIX) + - siJ..... [ sil) via discarding duplicated elements. We then discarded for every factor a E {0.1, 0.2.0........... the corresponding fraction of the features with highest (approximated) normalized intrinsic dimensionality from all data points. Following the selection process, the machine learning algorithms from the relevant papers applied to the cetture-reduced data sets using the same (hyper-) (byper-)parameter )arameeter configuration as the original. In order to facilitate an objective evaluation, the same scores (accuracy or fl) as those originally produced were collected over repeated training runs with ten different sceds.4 4 We did not test other feature selection methods as similar investigations were already done in Stubbemann et al. (2023b).
+
5.4 Observations
We present in this section the computational results and observations for the experiment. Here we focus on the details corresponding to the two research works GCN and SSGN+SLE. Afterwards we will state general obscrvations for the remaining experiments, but refer the reader to Appendix C for accompanying plots.
In Figure 3a we show the analysis for the intrinsic dimensionality of the three data sets from GCN. Because the data are differently sized we need to find a common representation. First we order the feature set for cach data set using the normalized intrinsic dimensionality of cach feature the score. On the x-axis we
4 An exception was the diffpool enzymes experiment, where only smaller number of runs was feasible given the runtime of the algorithm.
19

---

Published in Transactions on Machine Learning Research (10/2024)
give the relative position of the sorted feature set, i.e. position a indicates that the corresponding feature is at the sorted position o- |F]. As the measured normalized intrinsic dimensionality can vary widely between the data sets we decided to normalize it by dividing, for each data set D, the value O(D) by max, a(D), The corresponding values are depicted in the y-axis in Figure 3a.
We observe that all curves increase monotonically in value with respect to the ranked position of the fea- tures. This is expected as we sort by this value. However, the slope is solely dependent on the individual contributions of the features to the intrinsic dimension. The stair case pattern is not an artifact of the plot but rather results directly from the data set and its preprocessing. This indicates that lot of features have the same normalized intrinsic dimensionality per step.
We further observe that the pubmed data set (green) entails fealures with a similar high normalized intrinsic dimension. Or more general, the higher the line in the plot the more similar are the values of the individual features of a data set D compared to the maximal feature value maxy ((D) This allows for comparing the feature behavior of the different data sets. For example, with respect to this property we observe that the cora data set (orange) has more diverse distributed features compared to the pubmed or citeseer data set (blue).
Figure 4a demonstrates that the distributions for the normalized intrinsic dimension of the data sets used for the SAGN+SLE method are of greater variety than those discussed earlier. We can see that the cora data set does not have a prominent stair case pattern, which can be explained by different preprocessing steps that smooth the features relative to each other. One standout distribution is that of the reddit data set, which is shaped like hockey stick.
As we calculated the (approximated) normalized intrinsic dimensionality on the data sets, occasionally different normalized rankings for what seems to be the same data set emerged through different steps of their preprocessing. A highly visible example can be found in Appendix C with the reddit data set in the GraphSAGE (Figure 8a) and SGC (Figure 9a) experiments.
Accuracy and Intrinsic Dimension Figure 3b shows the accuracy of the resulting model when applying the GCN machine learning method to the feature reduced data sets. In this experiment, both the training and test data sets are feature-reduced. This ensures that the algorithm is trained and tested on the same set of features.
For each data set we reduced the number of features in steps of 1% up until 10% was reached. Here the percentage steps are taken with respect to the size of the complete feature set F. Afterwards we continued with 10% steps. Both are indicated on the x-axis in Figure 3b. After training the corresponding model we collected the obtained accuracy, similar to the original work. To achiove a meaningful estimate for the model behavior with regard to our dimensional data set perturbation, we measure the average over len ical runs with different seeds. The resulting standard deviation shown via error bars in the plot. Similar plots for the other papers can be found in Appendix C.
We observed that the methods sometimes failed to converge for smaller discarding values (< 0.01). This behavior was very irregular and we did not include these runs and their corresponding discarding values in the figures. Our investigation into the causes showed that this was usually due to some artifacts of the machine learning method, such as early stopping.
Additionally we conduct further experiments with random or reversed feature selection, where the latter means the discarding of features with the lowest normalized intrinsic dimensionality first. For the sake of completeness, by random we refer to the process of randomly selecting features from F. For all discarded data sets we applied the GCN method with ten different sceds. Due to the expected long run times, these extended experiments Grandom/reversee 1% discard steps) were not conducted for methods other than GCN.
For all data sets the resulting model performances are relatively stable under the aforementioned primary discarding method. Yet when applying the reverse selection method a fast deteriorating performance can be observed. This behavior starts already at the smallest discarded proportion and is very pronounced.
20

---

Published in Transactions on Machine Learning Research (10/2024)
10a
Pabased
10
a(D) (r] m/xy 0(D)s
0.7
10-2
Accuracy
10-3
0.6
10-4
chesser coterenee
200
10
0.0
0.2
0.4
0.6
0.8
1.0
010-2
10-1
Feature (relative ranked position)
Discarded Proportion
(a) Normalized intrinsic dimensionality (y-axis) of the cora, cilescer, and pubmed data sets plotted against relative ranked position of features (x-axis). For every data set the sorting key is defined by normalized intrinsic dimensionality divided by max ( The values themselves are normalized by the highest value and sorted in ascending order.
(b) Accuracy of the resulting model (y-axis) after al- tering the GCN data sets based on feature selection. We discarded fraction of features (x-axis) with the highest. normalized intrinsic dimensionality from the original data set. Curves labeled with random or versed used a random or reversed selection method re- spectively. Bars indicate standard deviation over ten repetitions with different seeds.
Figure 3: Influence of Intrinsic Dimension measured through feature selection for the GCN results.
A more intricate detail can be observed for the random discarding method by combining the information from the two Figures 3a and 3b. We find that the higher the line in Figure 3a the later (i.e., higher values of a) the break off between performance of normal and random discarding in Figure 3b.
We also see small fluctuations and drops in performance the highest discarding values. This becomes more apparent when directly visualizing the differences to the proposed discarding method.
This picture changes slightly when looking at the results related to the SAGN+SLF. experiments in Figure 4b. For some data sets the same stagnating behavior evident. For others, however, there is a marked drop in performance. Especially for the two ppi data sets there a greater variation in performance. Another different behavior can be seen for the yelp data set, where the performance starts decrease for lower discard factors.
5.5 Discussion
We now want to contextualize the observations and results. The Figures 3a and 4a show the distributions of normalized intrinsic dimensionality (NID) and we observe that distinct values arise for different data sets. We may note that the figures show relative and not absolute NID and therefore their respective values should not be compared. Furthermore, even the absence of this relative scaling, the mathematical modelling of the intrinsic dimension permits direct comparison.
For our discussion we compare the different values of NID to the performance of the corresponding models, shown in Figures 3b and 4b. From this we infer the following link. consider the difference between the lowest and the highest value of the NID for given data set. We find that this difference decreases in the following order: pubmed, citeseer, cora. If we look to the corresponding model performance in Figure 3b, we observe that the performance of the random feature selection divergences for different. proportions a. Interestingly this happens in the same order as before, albeit at different levels of accuracy. The difference between the lowest and highest values NID indicates that the individual contributions to the ID by the corresponding features more evenly distributed. For example, in the case of a horizontal line every feature contributes equally to the ID. Conversely, J-shape, observed in Figure 4a for the reddit data set.,
21

---

Published in Transactions on Machine Learning Research (10/2024)
10a
1.0
10-1
a(D) [r m/xyy a(D)
10-2
0.8
5000e
10
10
0.6
10-5
0.5
Bicks
10
POL
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Feature (relative ranked position)
Discarded Proportion
(a) Normalized intrinsic dimensionality plotted against relative ranked position of features. See Figure 3a for more detailed captions.
(b) Performance (measured by fl score) of resulting model after discarding features from data sets. See Figure 3b for more detailed captions.
Figure 4: Influence of Intrinsic Dimension measured through feature selection for the SAGN+SLE results.
indicates that a small number features is responsible for almost all of the ID value. Based on these deductions, we propose the following explanation. In a certain sense, features with a low NID can be usod by machine learning methods distinguish more data points. These features may allow a learming procedure to have more stable convergence, a shorter runtime, and higher final performance. In experimental study, we focus the interplay between the ID and the achieved model performance. At this point may note that all presented experiments are based on the same principal optimization task of stochastic gradient decent (SGD).
The observations concerning the shape of distribution of NID described above may permit to draw a connec- tion to the simplicity bias in neural networks (Arpit et al., 2017; Shah al., 2020; Valle-Pérez al., 2019), the tendency of SGD to find simple models. Although further research is needed to confirm this hypothesis, the proposed association would be consistent with the assumption that SGD weights features that carry more information higher.
In our random experiment, we discard uniformly from the set of features. In every step one may lose low and high dimensional features following the distributions as shown in Figures 3a and 4a. This means for a certain discarding proportion a there are almost no features with low NID available. Until those disappear, SGD has the possibility to use them for obtaining the objective. But when those "good" features are not included anymore the situation changes and the performance deteriorates rapidly. On the other hand if the features are discarded in order of decreasing NID then the inevitable deterioration of model performance be postponed for quite a bit. Conversely, when discarding features in a reverse order, e.g. ascending NID, the performance drops rapidly as the model has no access to those "good" features from the beginning.
However, we can not exclude the effect of artifacts of the method. Especially in the reverse case, where, for example, non-convergent behavior at the beginning of training triggers an early stopping condition that leads the abortion of the optimization routine. As we regard the methods as black boxes, we have not investigated these possibilities further.
Turning to Figure 4a, observe a few data sets that have a similar distribution of NID as those in Figure 3a. However, it seems that for some the contribution to the total NID are distributed more evenly among the features. This is particularly evident for the yelp data set. On the other hand, the distribution for the reddit data set much more extreme, where only a small set of features have extraordinary high contribution to overall NID. This figure also clearly shows the influence of preprocessing on the NID distribution. Whercas previously Figure 3a the line for the cora data set was clearly a step function, it has now become a much smoother slope. It seems that this has almost no influence the achieved model performance in both cases.
22

---

Published in Transactions on Machine Learning Research (10/2024)
The definitions of ID and NID indicate that these functions should have higher values for a data set than for any of its subsets, provided that the set of feature functions remains constant. This is clearly demonstrated in the graphs for both versions of the ppi data set, where one is the super-set of the other. It is also noteworthy that the final model performances for these data sets exhibit a high standard deviation, which is much higher than in any other experiments. It was not possible to determine the definite cause, but it seems reasonable to assume that some form of artifact of the method or mode collapse produced these high variations. The detailed figures for the remaining experiments can be found in the appendix.
We will now discuss the insights that can be gained from overarching analysis. To do this, we will use cross-method comparison, as only a few data sets have been processed by several papers. The aim is to identify similar distributions of NID and compare the effects on model performance. If both methods behave differently on these (possibly different) data sets, this could indicate that the NID has an effect on the methods.
The GraphSAGE and SAGN+SLE methods both use the reddil data set with the same preprocessing. The former shows slight deterioration, while the latter shows almost no change in model performance.
The SAGN+SLE method is applicd on the yelp data set, while the GCN and SGC methods are used on the pubmed data set. The NID distributions are quite similar, but the performance on the yelp data set continuously worsens with higher o, while the performance on the pubmed data set remains stable until the highest discarding proportions.
Both the GCN and SGC methods use the citeseer data set with the same preprocessing. The first method shows very stable performance, but the second method shows minor deterioration.
The GCN, SGC and SAGN+SLE methods all use the cora data set. Although the preprocessing is the same in GCN and SGC, there is a clear difference in performance.
The SSE and GraphSAGE method both use the large ppi data set mentioned carlier, but in both cases the performance deteriorates significantly, starting at different discarding proportions and speeds.
A possible alternative hypothesis for the above observations that the machine learning tasks at hand can be solved using only the information provided by the adjacency matrix of the graph data. In this case, the reduction of available node features would not affect the final performance. In contrast, observations from experiments with inverse feature discarding suggest that the previous statement may not be correct, as these observations showed a much larger degradation in model performance than the other way around.
In general, this highlights a limitation of the current. approach, as the chosen feature functions do not take graph edges into account. This indirectly related to the carlier discussion on the modelling choice of what to use as the underlying set for the geometric data set. We decided to use the node features as the base set X and ignored the edge features or the adjacency matrix. By using different modelling approach, the set of feature functions could be extended or constructed entirely differently.
Taken together, these comparisons provide a strong indication that there is an effect of NID on model performance. Although we have only used the NID as an auxiliary tool to measure the ID, it shows that different methods are influenced by the ID of the data set itself. However, it is difficult to quantify the extent of this dependence on the concentration phenomenon given the present experiments on these very different methods.
At this point it is necessary to go into more detail about the graphs for the experiments for the DiffPool model. The original code accompanying the DiffPool publication uses an one-hot encoding of node label as node attributes instead of the available original node attributes. This is not stated in the paper, and the results do not seem to be easily reproducible when switching to the conventional node attributes. Nevertheless, we made this change to make it compatible with the other experiments and our method. As a result, the graph of the accuracy achieved starts much lower for zero and low discard fractions a than the originally claimed performance would suggest. This also explaines why the results are not inffuenced by the discarding of features, as the node labels lack sufficient discriminatory power.
23

---

Published in Transactions on Machine Learning Research (10/2024)
5.6 Summarizing the Analysis and Limitations
We present overview of all experiments by combining the information about the intrinsic dimensionality and the model performance when features are discarded. To do this, we calculate the sun of the (approx- imate) normalized discrininability of the remaining features after discarding a fraction a. The value thus obtained can be normalized by the total sum of (approximate) normalized discriminability of all features. We perform this calculation for different values of a and all data scts. We plot these values against the achieved performance measure (accuracy or score) for the corresponding configuration. The results are depicted in Figure 5.
This leads to the striking observation that most models can cope with data sets reduced to about 30% of their original dimensionality without any loss of performance. Despite the large differences in the number of samples and features between the data sets, we do not observe a significant correlation between these features and the change in model performance.
Experiments with reversed feature selection (Figure 3) showed that there is an effect of the intrinsic dimension on the different learning methods. More specifically, we observed that this effect depends on the particular data set, as discussed in Subsection 5.5.
Our study is limited in various aspects, which we will discuss below. We observe in our investigation a low frequeney of overlapping use of data sets, which is a consequence of the selection process and the underlying requirements for reproducibility. Consequently, it is challenging to compare different methods on the dimensionally reduced data sets. However, the data sets cileseer, cora, pubmed and reddät have non-trivial support over the considered papers. In this regard, we observe a similar pattern of behaviour as previously described (cf. Figure 5).
Furthermore our results build upon a rather small set of selected candidates. This could be tackled with allocating more time for achieving reproducibility per paper, which would allow for fixing or circumvent reproducibility barriers by searching for the right combination of technical tricks. It might also be possible to apply the individual methods from the publications to the other data sets as well.
The proposed method only indirectly measures the effects of the concentration of measure phenomenon through the perspective of the geometric intrinsic dimension. Morcover, is is not possible to provide an extensive overview of the complete ID influence. This is due to the fact that is unclear if the ML methods can draw more (complicated) feature functions than the considered ones in their processing of the data. In this case, the current restriction would be hindrance to measuring the true impact of the ID on the methods. However, for the function class in question, we can rely on the guaranteed computability. Upon reflection, it may be not necessary to do so, as even within this restricted scope, an influence could be demonstrated.
Furthermore, the present work does not include a comparison with other approaches for measuring the influ- ence of dimensions and any feature selection methods based on them. It is not clear if those methods would measure the same properties of the ML methods given their different underlying theoretical frameworks.
The experiments indicate that the distribution of dimensionality has a detrimental impact on the model performance. However, further experiments are required to gain a comprehensive understanding of this rela- tionship. In particular, the measured ID is contingent upon the features functions employed. Consequently, by utilizing more comprehensive function classes, the intrinsic dimension can be more accurately determined.
Finally, we want to explain the rationale behind the decision not to utilise the correlation coefficient or alternative methodology to quantify the direct correspondence between intrinsic dimension and model accuracy in order to facilitate a comparison of data sets. In order to do so, will point out two theoretical aspects. Firstly, different data sets will exhibit disparate (absolute) diameters, even when employing the same feature functions. Secondly, the observable diameter is linear with regard to the feature functions, e.g. rObservableDiameter// (D) = DbsserabbleDiameter((>)) where 7D = (X, [fr : x + 7f(x) I E F), p), for T E R (see Hanika et al. (2022)). Thus, to consider their ratios is less informative without normalizing with regard to these properties. Therefore a first meaningful way to compare values is to focus on the ordinal relation of the feature attributes with regard to intrinsic dimension. Altogether, the experiments
24

---

Published in Transactions on Machine Learning Research (10/2024)
1.0
0.8
0.8
Accuracy
score
(0-ppeage-44))
(dee)
mvion (1a7)
fl
rediit (gmphaggonuug
pol.nneg (regs) (00)
0.5
(eteclensl-gre) HAte teclatieoasagrn casynen (41d)poo)))
polone Carn ait (elatoonaogge)
prlhoned (ep1)
0.4
0.0
0.3 0.1
0.6 0.7
1.0
0.0
0.1
0.2
0.4
0.6
0.7
1.0
normalized sum selected a(D),
normalized sum of selected 0(D),
(a) Accuracy of data set and paper combinations
(b) fl scores of data set and paper combinations
Figure 5: Overview of evaluation of data set and paper combinations over the remaining intrinsic dimen- sionality. The x-axis is the sum of the (approximated) normalized intrinsic dimensionality of the remaining features normalized by the total sum for the whole feature set. The y-axis is the resulting evaluation score obtained by the method trained on the data set with corresponding feature selection.
already show a connection between feature discarding and model performance and highlight that the order of discarding matters.
6 Reproducibility of the Presented Work
As publication about reproducibility, it only fair also consider the own reproducibility as well. In general the reproducibility of our work is limited by the reproducibility of the used papers. We rely exclusively data sets provided by them and our source code is based the one published by them. For the purposes of our experiments, it was not necessary to modify the hyperparameters of the models. In order to achieve a high degree of reproducibility, we have provided the individual scripts for the reproducibility and dimensionality experiments. These scripts include the explicitly specified environments that were used and the necessary changes the original source code. addition, we have endeavoured to adhere the general guidelines set out by (Pineau et al., 2021; paperswitheode, 2021), as well as our own ontology. In total, the experiments produced approximately 600 models (6 papers, 10 factors, 10 seeds). It is not feasible to provide weights for all of them, especially if the original source code did not include sophisticated management of the paths where checkpoints were stored. Given the special form of our publication, we do not provide the model weights. The complete resources for source code, logs, and results of our experiments can be found at
org/ httpa///...rr///...../////eoooo. 107279077
7 Conclusions
In this work, we presented an ontology to investigate the reproducibility of machine learning research. This ontology can be used evaluate the reproducibility of scientific publications standardized manner. For this, we assume that scientific evidence can be generaled via theoretical evidence (for example via theorems and proofs) or via empirical evidence provided by scientific experiments. In machine learning, reproducibility research mainly focuses on the extent to which other researchers can re-execute the experiments with the same results. Consequently, our work was primarily concerned with empirical evidence, for which we have proposed a set of attributes for evaluating the level of reprodueibility a specific work. These attributes can divided into three categories: data set, software and computational result.
Once reprodueibility has been established, the next step identify the relevant influential factors that affect the outcome of the experiments. One such factor that has been the subject of the presented inves-
25

---

Published in Transactions on Machine Learning Research (10/2024)
tigation is the intrinsic dimension, which measures the inffuence of the curse of dimensionality on given data set. Tn the second part of our work, we investigated to what extent the results of empirical experiments depend on the intrinsic dimension of the data sets used for training. To provide further detail, we examined whether alterations to the intrinsic dimensionality of the employed data sets were associated with changes in model performance.
To illustrate the practical application of our ontology, we have provided detailed descriptions of how the attributes of our ontology be used to evaluate the reproducibility of research on graph neural networks. Furthermore, we have investigated which of the well-known methods are not affected by modifications of the intrinsic dimensionality of the data sets on which they are trained on.
Our investigation indicates that attributes of reproducibility, which are part of the dala sel calegory of our ontology, are similarly addressed in the majority of work on graph neural networks. However, we did identify notable differences between the various methods with regard to the extend to which the data sets are documented within the README file. The quality and quantity of the documentation emerged as a crucial factors in successful reproduction with reasonable effort. The documentation of specific work is encompassed by our ontology within the category software. In general, the software category allowed for an accurate differentiation of the different papers with regard to their reproducibility. Attributes in the category compulational resull can be divided into two distinct groups. The first group comprises weakly separating attributes, which either represent basic reproducibility rules that are fulfilled by all of the investigated methods or strict requirements that none the methods satisfy. The second group encompasses attributes that enable the distinction of the various methods, as these attributes are only covered by a subset of the investigated methods.
The findings presented in the second part of our study provide compelling evidence that the ID has influence on ML algorithms. To be more detailed, our experiments show that dropping features with high individual ID has a varying impact on model performance. For example, for the GCN method, dropping high-dimensional features does not fundamentally decrease accuracies. In contrast, dropping these features when learning with GraphSAGR leads to stronger deterioration of performance. Conversely, starting with the removal of features with low individual ID results in more pronounced performance declines across all methods. This suggests that the graph learning approaches currently in use are vulncrable to changes in intrinsic dimensionality, while their resilience to the elimination of non-discriminative features (i.e., features with high individual ID) varies considerably. It has been demonstrated that the ID can be employed as a means of measuring this robustness. However, further research is required in order to ascertain the transferability of this insight. It is possible that the ID may prove to be a candidate for another meaningful attribute within the presented ontology of reproducibility.
8 Recommendations
Based on our findings, we recommend the following points as best practice for reproducible machine learning research.
Write one main script that does everything necessary. That includes setup and or tear- down preparation of the compute environment, downloading and preprocessing of data sets, and running of all reported experiments. Tt might be beneficial to chose an existing software package when handling larger data or when working on compute clusters.
Log all relevant information into files. This encompasses all outcomes, utilised input variables and also selected hyperparameters, in the event that form of hyperparameter optimisation was conducted. Often, such information is only printed to the output terminal, which hinders repro- ducibility.
Store chockpoints of all obtained models. As multiple runs with only different seeds are required, it is imperative that the wrapping script/softwan ensures that no computed checkpoint is overwritten in the subsequent iteration.
26

---

Published in Transactions on Machine Learning Research (10/2024)
A workflow management systems is helpful for automated aggregation and evaluation of outputs and creation of visualizations. The reproducibility of research paper is significantly enhanced when all steps from the experiment to the final paper, including reported results, tables, and displayed figures, can be automatically created by a dedicated script that is part of the publishod code. The full procedure can easily be implemented via workflow management system.
Furthermore, we give three recommendations on how to account for the concept of intrinsic dimensionality in the scope of machine learning research.
Consider the intrinsic dimensionality of the used training data sets when comparing machine learning algorithms. Our experiments indicate that well-cstablished graph neural net- work approaches are significantly impaired by the increasing intrinsic dimensionality of the input data. Consequently, when comparing their performance, is of paramount importance to ascertain the ID of the used data in order to accurately assess the extent to which performance differences are attributable to the curse of dimensionality.
Investigate, if discarding of high-dimensional features is possible for the chosen GNN. It is possible to discard significant fraction of features with high normalized intrinsic dimensionality from certain graph neural network models without a fundamental drop in performance. For example, when considcring the GCN model, it is possible to drop up to 70% of the total mumber of features without decrease accuracy, while this not possible for GraphSAGE.
Account for transformations of the NID-distributions induced via preprocessing. The preprocessing of features lypically incorporates global interactions between them (i.e., averages), which alters the NID distribution in non-trivial manner. One such case was observed in the reddit data set, where both SAGN+SLE and SGC applicd distinct preprocessing procedures, resulting in markedly disparate NID distributions. As previously stated, this alteration of the NID distributions can profoundly impact model performance.
9 Limitations and Future Work
The current study is subject to certain limitations, which will be addressed in future research. Firstly, as discussed in Subsection 4.3, our ontology is limited to fixed granularity. It has become evident that the varying depth of analysis required for reproducibility cannot be represented by the proposed ontology. Consequently, we will develop a hierarchy of ontologies with different levels of granularity by further splitting or aggregating the current attributes. For instance, in the event that a given paper presents a multitude of distinct experiments, it may be prudent to redefine altributes R2,R3 and R4 in order to encompass a broader spectrum of error ranges and fractions of reproducoeee outcomes.
Secondly, the current study is focused in Section 2, variely of different ID estimators exist. Consequently, future work will investigate whether the reported observations are applicable to different concepts for intrinsic dimensionality.
specific concept of intrinsic dimensionality. However, discussed
Thirdly, our approach considers only intrinsic dimensionality for a specific feature set, namely the usual coordinate projections. However, different machine learning methods may incorporate different aspects of the data. Consequently, future work will investigate how these different aspects can be formalized as feature functions. This will lead to an ID) not on for models instead of data sets. Here, the crucial problem will be the identification of finite and computational feasible feature set which captures the model behavior.
Acknowledgment
The authors thank the State of Hesse, Germany for funding this work as part of the LOEWE Exploration project "Dimension Curse Detector" under grant
27

---

Published in Transactions on Machine Learning Research (10/2024)
References
Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Hrayr Harutyunyan, N. Alipourfard, Kristina Lerman, G. V. Steeg, and A. Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. ArXiv, abs/1105.00067:2) 29, 4 2019. URL
wtt/p/f/gg/p///ee////t//pgg 00067.
Ekin Akyürek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. Tracing knowledge in language models back to the training data. ArXiv, abs/2205.11482 5 2022. URL
https://arxiv. http:////aaxvv...//ppp... 2205/ 12122220222..........1 11482.
Allen Institute for Artificial Intelligence. Semantic Scholar, 2022. URL http::/www..emmtticcchho.. org/ [Online; accessed 2023-12-11..
Alessio Ansuini, Alessandro Laio, Jakob Н Macke, and Davide Zoccolan. Intrinsic dimension of data rep- resentations in deep neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-uuc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL attps://proceeddng www.neuuu...c.aappeeeeeee. http://ppppeeeeeeeeppp////////// (tce000110cc30011eeeedddddbbbbb3 pdf.
Devansh Arpit, Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kan- wal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization deep networks. In Doina Precup and Yee Whye Teh (eds.), International conference on machine learn- ing, volume 70, pp. 233 242. PMLR, JMLR.org, 8 2017. URL arpiti7aa/rpptt77..dtt
ashleve and Contributors. Lightning Hydra Template, 2022. URL http::://gittuub..om/aaaatt
-eggttinn--yddrr-temlaaaa [Online; accessed 2023-11-22]
Deyu Bо, Xiao Wang, Chuan Shi, and Huawei Shen. Beyond low-frequency information in graph convolu- tional networks. In AAAI Conference on Artificial Intelligence, volume 35, pp. 3950-3957, Association for the Advancement of Artificial Intelligence (AAAI), 5 2021. doi: 10.1609/ /aa//aaaai....... attps://arxiv: owtp:///rrp/pp/////// 00797.
URL
Xavier Bouthillier, César Laurent, and Pascal Vincent. Unreprodueible research is reproducible. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), International Conference on Machine Learning, volume 97, pp. 725 734. PMLR, 5 2019.
Arturo Casadevall and Ferric C. Fang. Reproducible science. Infection and Immunity, 78(12)44972 4975, 12 2010. ISSN 0019-9567. URL sttps://europpe
Boyuan Chen, Mingzhi Wen, Yong Shi, Dayi Lin, Gopi Krishnan Rajbahadur, and Zhen Ming Jiang. Towards training reproducible deep learning models. 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE), pp. 2202 2214, 5 2022. doi:
351166..
Christian Collberg, Todd Proebsting, and Alex M Warren. Repeatability and benefaction in computer systems research. University of Arizona TR, 14(4), 2015.
Gabriele Corso, Luca Cavalleri, D. Beaini, P. Lio', and Petar Velickovic. Principal neighbourhood aggregation for graph nets. ArXiv, abs//2004055788, 4 2020. URL https:
Ailin Deng and Bryan Hooi. Graph neural network- based anomaly detection in multivariate time series. 2021. ISSN 2374-3468. doi: 10.1609/aami 23..0///1ii.3355.65222., URL https:
ArXiv, abs//200..09944::27---233 // wwwpxio.rrg/aff//1......oo
Diego Doino, Aldo Glielmo, Alessio Ansuini, and Alessandro Laio. Hierarchical nucleation in deep neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (ods.). Advances Neural Information Processing Systems, volume 33, pp. 7526-7536. Curran Associates, Inc., 2020. URL https: https:::/proceddigg..oooo wwtpaperrccpppp.r..cpmm httpp//pe..ieeeeeppr//////////// B442b444300r02at55599999ffttt Paper. pdf.
28

---

Published in Transactions on Machine Learning Research (10/2024)
Yifan Feng, Haoxuan You, Zizhao Zhang, R. Ji, and Yue Gao. Hypergraph neural networks. In AAAI Conference on Artificial Intelligence, pp. 3558 3565, 9 2018. doi: 10.1609/aami.v3310 33330...3301133333355 URL
https://ojs.aaai org/index. httpa//pppp/////aatiii////////// http/A////arriiae///mmm////////
Damien François, Vincent Wertz, and Michel Verleysen. The concentration of fractional distances. 1EEE
Transactions on Knowledge and Data Engineering, 19(7):873 886, 2007. ISSN 1041-4347. doi: 10.1109/
tkde:2007.1037.
Juliana Freire, Philippe Bonnet, and Dennis Shasha. Computational reproducibility: state-of-the-art, chal- lenges, and database research opportunitics. Proceedings of the 2012 ACM SIGMOD International Con- ference on Management of Data, 5 2012. doi: 10.1145/ 10.1145/221383... 22//2222211222122 2213908.
Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King. Magun: Metapath aggregated graph neural network for heterogeneous graph embedding. Proceedings of The Web Conference 2020, 2020. doi: 10.1145/
3366423.3380297. URL https: http:///rxxxoooo/paf..//// httpp///000.0100 01680.
Bernhard Ganter and Rudolf Wille. Formal Concept Analysis: Mathematical Foundations. Springer- Verlag, Berlin, Heidelberg, 1st edition, 1997. ISBN 3540627715. doi:
Jean Golay, Michael Leuenberger, and Mikhail F. Kanevski. Feature selection for regression problems based on the morisita estimator of intrinsic dimension: Concept and case studies. Pattern Recognit., 70:126 1388 1 2016.
Steven N. Goodman, Daniele Fanelli, and John P. A. Ioannidis. What does research reproducibility mean? Science Translational Medicine, 84341):341ps12 341ps12, 6 2016. ISSN 1946-6234.
Mikhael Gromov and Vitali Milman. A topological application of the isoperimetric inequality. American Journal of Mathematics, 145(4): 854, 8 1983. ISSN 0002-9327. doi: 40.2307/237422)
Odd Erik Gundersen. The fundamental principles of reproducibility. ArXiv, abs/ 2bs/2011.10098,, 11 2020.
Odd Erik Gundersen and Sigbjørn Kjensmo. State of the art: Reproducibility in artificial intelligence. In Sheila A. Mcllraith and Kilian Q Weinberger (eds.), AAAI Conference on Artificial Intelligence, vol- ume 32. Association for the Advancement of Artificial Intelligence (AAAI), 2018. doi: 10.1609/aad
+32i1.11500 URL. attps://ojs aaai /erg//nde....//
Odd Erik Gundersen, Yolanda Gil, and David W. Aha. On reproducible ai: Towards reprodueible research, open seience, and digital scholarship in ai publications. Al Mag., 68, 2018. ISSN 0738-4602. URL https://www wwai. maaaiiorg/o////inn//ee
Odd Erik Gundersen, Kevin Coakley, and Christine R. Kirkpatrick. Sources of irreprodueibilitty in machine learning: A review. ArXiu, abs/ Abs/2204.07610, 4 2022a. ISSN 2331-8422. doi: 2448550/arxxv.2244...
Odd Erik Gundersen, Sacid Shamsalici, and Richard Juul Isdahl. Do machine learning platforms provide out-of-the-box reproducibility? Future Gener. Comput. Syst., 126:34 47, 1 2022b. ISSN 0167-739X. doi: f0.1016/j.future22000 20w.:rree.22...000011 .06.014.
William L.. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Tn Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), NIPS, volume 30, pp. 1024-1034, 6 2017. URL https: httpse//papers .iis. cc/ on oa-ger graphs .df..
Thomas Hamm and Ingo Steinwart. Adaptive learning rates for support vector machines working on data with low intrinsic dimension. The Annals of Statistics, 3 2020. doi: 10.48550//rxiv.20033 http://awrxxx.....22000000 206..020... URL
https://arxiv.. org/pdf/ /2003. 06202.
Zayd Hammoudeh and Daniel Lowd. Training data influence analysis and estimation: A survey. ArXiv, abs/2212.04411, 12 2022. doi: (445550/arxiv.212200000 URL htttp::/arxiv.orgg/ /2212//./2221211112
29

---

Published in Transactions on Machine Learning Research (10/2024)
Tom Hanika, Friedrich Martin Schneider, and Gerd Stumme. Intrinsic dimension of geometric data sets. Tohoku Mathematical Journal, 74(3): 52, 3 2022. ISSN 0040-8735. doi: 10.2748 1127748/mm..202011113 URL
http:///xxvv..rrggo /t/0p//../..99.5.
Michael Hazoglu, Vivek Kulkarni, Steven S. Skiena, and Ken A. Dill. Citation histories of papers: sometimes the rich get richer, sometimes they don't. ArXio, abs/1703.04444 3 2017. ISSN 2331-8422.
Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightenn: Simplifying and powering graph convolution network for recommendation. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2 2020. doi: 10. 10.1145/3397271. 3401063. URL https://arxiv.. org/ 2002./200..22222
Matthias Hein and Jean- Audibert. Intrinsic dimensionality estimation of submanifolds in rd. Proceedings of the 22nd international conference on Machine learning, 2005.
D. Hong, Lianru Gao, Jing Yao, Bing Zhang, A. Plaza, and J. Chamussot. Graph convolutional networks for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 00-5966-5978 8 2020. ISSN 0196-2892. doi:
URL attps://arxit http://xxivv.ogppttt///////s//// /2008. 22000000070.0
Michael E. Houle. Dimensionalityy, discriminability, density and distance distributions. 2013 IEEE 13th International Conference on Data Mining Workshops, pp. 168-473, 12 2013. doi: 10. 10.1109/icdmw. 2013.131
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. ArXiv, abs/2005.00687, 5 2020a. doi: wtt.500//axxv.205.00060
Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yinhou Sun. Heterogeneous graph transformer. Proceedings of The Web Conference 2020, 4 2020b. doi: pdf/10. 1444//3643333623 3380027.
URL sttps://dl htcp///llcc..cc////////////./. acm. org/ /
ICLR. Reproducibility workshop series, 2019.
URL http:://ittee.ggggle..o////////
icml poor-ropooocciilllyyyyyrrrraaaa [Online; accessed 2023-11-15)
Peter Ivie and Douglas Thain. Reproducibility in scientific computing. ACM Computing Surveys (CSUR), 51(3):1 36, 7 2018. ISSN 0360-0300. doi: 10.1145/ 3186666.
Wei Jin, Yao Ma, Xiaorui Liu, Xianfeng Tang, Suhang Wang, and Jiliang Tang. Graph structure learning for robust graph neural networks. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Dala Mining, 2020. doi: N0.1145/339444863300083 org/doi/pdf/100 10. 1145/3394486 3403049.
URL https://dl.aa acm.
Dongkwan Kim and Alice H. Oh. How to find your friendly neighborhood: Graph attention design with self-supervision. Ar abs/2204.04879, 4 2022. doi: 10.48550/arxiv 220.000....0. URL https://axxiv.
org/pdf/ 2004f//20440487
Jisu Kim, Alessandro Rinaldo, and Larry A. Wasserman. Minimax rates for estimating the dimension of a manifold. J. Comput. Geom., 10:42 95, 5 2016. ISSN 1920-180X. doi: 10.20382/jocg URL
https://inria. hww.://ww..aa......eee/eeeeeeeee secieemce/hal-0225566444
Thomas Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. ArXiv, hhss/160..02001 9 2016. URL https://export, arxiv. sekiv..rr//dde www.ooogg////oooo..ooooo http//df/600....///// 02907
Justin Kitzes, Daniel Turek, and Fatma Deniz. The practice of reproducible research: case studies and lessons from the data-intensive sciences. Univ California Press, 2018
Johannes Klicpera, Alcksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks moet personalized pagerank. In International Conference on Learning Representations, 2018. URL https:// http:rrrrieeeett///addddd--gggg 2A9Ym.
30

---

Published in Transactions on Machine Learning Research (10/2024)
Pang Wei Koh and Percy Liang. Understanding black-box predictions via. inftuence functions. In Doina Precup and Yee Whye Teh (eds.), International Conference Machine Learning, volume 70, pp. 1885- 1894. JMLR.org, 3 2017. URL https://proceeedngss http:/eeiiigg.......rrss///////s htt../ppee///////oo11///////111 pdf.
Burak Koçak, Ece Ateg, Emine Sebnem Durmaz, Melis Baykara Ulusan, and Ozgur Kilickesmez. Influence segmentation margin on machine learning-basee high-dimensional quantitative ct texture analysis: a reproducibility study on renal clear cell carcinomas. European Radiology, 29(9):1-11, 2019. TSSN 0938- 7994. doi: 000007/000330001--00333
Aleksandr Laptev, Roman Korostik, Aleksey Svischev, Andrei Andrusenko, Ivan Medennikov, and Sergey V. Rybin. You do not need more data: Improving end-to-end speech recognition by text-to-speech data augmentation. 2020 13th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEN) pp. 439-444, 10 2020. doi: 10.1109 http://pmmmmm113.....2. Ttw.:/63...220.222356 URL ottpadf//g//////////ee/ http://arxxxooo/pp//////// attps://arxiv.oom
07157.
Fabian Latorre, Leello Tadesse Dadi, Paul Rolland, and Volkan Cevher. The effect of the intrinsic dimension on the generalization of quadratic classifiers. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (ods.), Neural Information Processing Systems, volume 34, 12 2021. URL httts:://infocciennc.ooo wpw..cchreccrd//29655
Junhyun Lee, Tnyeop Lee, and Jaewoo Kang. Self-attention graph pooling. Arxi abs/ 0001199400802, 4 2019. URL
190./1994..00 080.0. pdf.
G. Li, Matthias Müller, Ali K. Thabet, and Bernard Ghanem. Deepgens: Can gens go deep cnns? 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9266 2265-9275, 10 2019. doi: 10.1 http://ccvv.22..009090 URL https: http:///a.xvoo//paaa//ae org/p/t/ 03751.
Lightning AT and Contributors. Lightning, 2022. URL (Online; accessed 2023-11-22].
Chao Liu, Cuiyun Gao, Xin Xia, David Lo, John C. Grundy, and Xiaohu Yang. On the replicability and reproducibility of deep learning in software engineering. ArXiv, abs/2006.14244,, 6 2020a. doi: 10.1145/ 3477535.
Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards deeper graph neural networks. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Adilya Prakash (eds.), Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 338 348. ACM, 8 2020b. doi: 10.1145/ 10.1145/3394488 /3394486. 3403076. URL https://dii htc.//.lllccmmm.////mm..m/mmm acm. wwt.laammooo//oooo prg/doi/pdff///e// 11/111111111/444444430340300
Diego Marchoggianí and Ivan Titov. Encoding sentences with graph convolutional networks for semantic role labeling. ArXiu, abs/1703.04826, 3 2017. doi: 10.18633/11/d17-1133 URL https://arxit: org/
0t//1773..04226 pdf.
Vitali Milman. The heritage of p. lévy in geometrical functional analysis. Astérisque, 157((58):273 301, 1988.
Vitali Milman. Topics in asymptotic geometric analysis. In Visions in Mathematics, pp. 192-815. Springer, 2000. ISBN 978-3034604246. doi: 10.100779988333344--0022 8.
Dengyao Mo and Samuel H. Huang. Fractal-based intrinsic dimension estimation and its application in dimensionality reduction. IEEE Transactions on Knowledge and Data Engineering, 24(1):59 71, 2012. ISSN 1041-4347. doi: 2.1109/tkee:200
Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, J. E. Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In AAAT Conference on Artificial Intelligence, pp. 4602-4609. Cornell University, 10 2018. doi: 10.1609/aaai, URL https://export, arxiv..rggaa...... http:/ddf1811.0222/
31

---

Published in Transactions on Machine Learning Research (10/2024)
National Academies of Sciences, Engincering and Medicine.
Reproducibility and Replicability
in Science. The National Academies Press, Washington, DC, 9 2019. ISBN 978-0-309-
48616-3. doi: 10.17226/25303. URL attps://nas nationalacademies wttpeaaawwnnes.oerraaaaaaalaaaae org/catalog/26303)
Nature Special. Challenges in irreproducible research, 2018. URL prbfkwwwvz/. [Ouline; accessed 2023-11-15].
paperswithcode. Tips for Publishing Research Code, 2021. URL http:://gtthuu...mmpppppsttttttt seleassng-research-com [Online; accessed 2023-11-15].
Hongbin Pei, Bingzhen Wei, K. Chang, Yu Lei, and Bo Yang. Geom-gen: Geometric graph convolutional networks. ArXiv, abs/2002.05287, 2020. URL http:://araav.orr//o/////000 05287.
Roger D. Peng. Reproducible research in computational seience. Science, (3:6600)) 1227, 12 2011. ISSN 0036-8075. URL ptttps:/eeurppeee org org/ /ttp://cceeeeeeme/a//0////rreere
Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification using deep learning. ArXiv, abs/1712.046211, 12 2017. URL http::////axiv.oo/pp//./.. 11tpp//.11...11111111111 pdf.
Vladimir Pestov. On the geometry of similarity search: Dimensionality curse and concentration of measure. Inf. Process. Lett., 73:47-51, 1 1999. doi: 10..000/500200001099))0111133310 URL https://arxiv.. orr// pss/cs/// /9001000vv
Vladimir Pestov. An axiomatic approach to intrinsic dimension of a dataset. Newral networks the official journal of the International Neural Network Society, 214-33):044-13,, 12 2007a. doi: 10.1016/j.neunne. 2007.12.030. URL
http://aaxxv...opddtee /0712 2063v1.
Vladimir Pestov. Intrinsic dimension of a dataset: what properties does one expect? 2007 International Joint Conference on Neural Networks, pp. 2959-2964, 8 2007b. doi: 10.1109/jcmm.2007.4777 20.7.03077..... URL
Vladimir Pestov. Indexability, concentration, and ve theory. In J. Discrete Algorithms, volume 13, pp. 2-18.
ACM, 2010a. doi: 10.1145/180234411622366
Vladimir Pestov. Intrinsic dimensionality. ArXiv, abs/ Abs/1007.5318.8 11, 2010b. ISSN 1946-7729. doi:
10.1145/1862413. 1862416. URL
5318.
Vladimir Pestov. Is the kk-nn classifier in high dimensions affected by the curse of dimensionality? Comput. Math. Appl., 65:1427 1437, 2011. doi: 10.1016/j 1..1016/..caww.222. Nammw.2012.09.00
Hung Viet Pham, Shangshu Qian, Jiannan Wang, Thibaud Lutellier, Jonathan Rosenthal, Lin Tan, Yaoliang Yu, and Nachiappan Nagappan. Problems and opportumities in training deep learning software systems: An analysis of variance. 2020 35th IEEE/ACA AACM/ International Conference Automated Software Engineering doi: 0..1145/3324884.22
(ASE), pp. 771-783, 9 2020.
Joelle Pincau. The machine learning reproducibility checklist, v2.0, 2020. URL https://www.cs.nog - http:p/nwaw/.pppppoo..ooooeeeeee pdf. [Online; accessed 2023-11-15)
Joelle Pineau, Philippe Vincent-Lamarry Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence
d' Alché Buc, Emily B. Fox, and H. Larochelle. Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program). JMLR, 22(164):1-20 2021. URL http:
http://.oo//pppeaaaeeee wttp:/apppes//22203333 html.
Hans Ekkehard Plesser. Reproducibility vs. replicability: A brief history of confused terminology. Frontiers
in Neuroinformatics, 11, 2018. ISSN 1662-5196. doi: 10.3389 1089//finff20000 2017.00076. URL tetps://www.
irentierssn..rg/arrtiici
p017
32

---

Published in Transactions on Machine Learning Research (10/2024)
Phillip E. Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrin-
sic dimension of images and its impact on learning. ArXin, (08/2104,08894, 2021. URL https:
//export.arxize arxiv.ovvoo........... wttppo////.....0.//..
Derek Price. Networks of scientific papers. Science, 149 3683(3663):510 5, 7 1965. ISSN 0036-8075.
Garima Pruthi, Frederick Liu, Mukund Sundararajan, and Satyen Kale. Estimating training data influence by tracking gradient descent. ArXin, abs/ 00s/2002.08484, 2 2020. URL https://export arxiv. org/pdft 2002.08484.
Gustavo Correa Publio, Diego Esteves, Agnieszka Lawrynowiez, P. Panov, Larisa N. Soldatova, Tommaso Soru, Joaquin Vanschoren, and Hamid Zafar. MI-schema: Exposing the semantics of machine learning with schemas and ontologies. ArXiv, abs/1807.05335 7 2018.
J. Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gee: Graph contrastive coding for graph neural network pre-training. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 6 2020. doi: 0111145/339446.3440000 3403168.
URL attps://arxiv.. http://axxvv..gg/pee// 0200000006.0.......
Edward Raff. A stop toward quantifying independently reproducible machine learning rescarch. ArXiv, abs/1909.06674, 2019.
ReScience C. Rescience journal, 2023. URL http://ee//reecccncccc
[Online; accessed 2023-09-09].
Justin Salamon and Juan Pablo Bello. Deep convolutional neural networks and data augmentation for environmental sound classification. IEEE Signal Processing Letters, 24:279-288 8 2016. doi: 10.1109/lsp. 2017.2657381.
Victor Garcia Satorras and Joan Bruna. Few-shot learning with graph neural networks. abs/1711.04043, 11 2017. URL https://expps: arxiv. warg/ddf/ii 1ttff//111.00000
Ar
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling, E (n) equivariant graph neural networks. Marina Meila and Tong Zhang (eds.), International conference on machine learning, volume 139, pp. 9323 9332. PMLR, PMLR, 2021.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling, Modeling relational data with graph convolutional networks. In Aldo Gangemi, Roberto Navigli, Maria- Esther Vidal, Pascal Hitzler, Raphaël Troncy, Laura Hollink, Anna Tordai, and Mehwish Alam (eds.); The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3 7, 2018, Proceedings 15, volume 10843, pp. 593 607. Springer, Springer Science+ Media, 2018. ISBN 978-3319934167. doi: 10.1007 10.1007/978-30 197799-377------- 11--319-9341111 904---34774--101 38. URL attps://researchh vu, nlw/////fille////////// pdf.
bata_with_
Harshay Shah, Kaustav Tamuly, Aditi Raghumathan, Pratock Jain, and Pranceth Netrapalli. The pitfalls of simplicity bias in neural networks, 6 2020. URL https: htapp:://proceedingg. htt.:eeeeddinnn...i............. pdf.
Mostafa Shahriari, Rudolf Ramler, and Lukas Fischer. How do deep-learning framework versions affect the reproducibility of neural network modcls? Machine Learning and Knowledge Extraction, 4(4):888-911, 2022. ISSN 2504-4990. doi: 10.3390 333390/make44400455, URL https: : http///ww.... hww../dpp..c///2244444949///////
pdd2vversion==1655756655
M. Simonovsky and N. Komodakis. Dynamic edge-conditioned filters in convolutional neural networks on graphs. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 29 38, 7 2017. ISSN 1063-6919. doi: 10.1109/cvpia /vpp/ URL https: ttops:///rxxia h/taxaxxvvorrr/ wwt.ooggg//////.ooo /170//4f/111/...////////////// 02901.
Koustuv Sinha and Jessica Zosa Forde. ML Reprodueibility Tools and Best Practices, 2020. URL https:
//koustuvsinha.cc http://wwwwwwwwwwwcwewweeeeeee.e cow/practices_ae....mm for preprroduuuuiiiity...... hrtpraatciceeffoorrrrrrcciiibttt
[Online; accessed 2023-11-15).
33

---

Published in Transactions on Machine Learning Research (10/2024)
Springer Nature. Reporting standards and availability of data, materials, code and protocols. Nature, 2020. URL accessed 2023-11-15).
[Online:
Victoria Stodden, David H. Bailey, Jonathan Michael Borwein, Randall J. LeVeque, William Rider, and W. A. Stein. Setting the default to reproducible reproducibility in computational and experimental math- ematics. ICERM Workshop, 46, 2013.
Maximilian Stubbemann, Tom Hanika, and Friedrich Martin Schneider. Intrinsic dimension for large-scale geometric learning. Transactions on Machine Learning Research, 2023, 2023a. URL https://openrevierr ner/forrmm?dd=5BBDD0MMM
Maximilian Stubbemann, Tobias Hille, and Tom Hanika. Selecting features by their resilience the curse of dimensionality. CoRR, abs/2304.004555, 2023b. doi: 10.48550/ (005550//RXXV
Chuxiong Sun and Guoshi Wu. Scalable and adaptive graph neural networks with selfllbee-enhanced train- ing. ArXiv, abs/ 29977664 4 2021. ISSN 2331-8422. URL https://exxort..arrte arxiv, wttp://wwwww.wooo..oo////./. 09376.
Divya Suryakumar, Andrew H. Sung, and Qingzhong Liu. Influence of machine learning vs. ranking algorithm on the critical dimension. International Journal of Future Computer and Communication, pp. 215-219, 2013. ISSN 2010-3751. doi: 77763/jjfcc.2113.22
Vasilis Syrgkanis and Manolis Zampetakis. Estimation and inference with trees and forests in high dimen- sions. In Jacob Abernethy and Shivani Agarwal (eds.), Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pp. 3453 3454. PMLR, 09 12 Jul 2020. html.
URL https: /ppp://///cceeng.... mml... wtt.:eeeiinnn......eese///////// wttpss//t///ssssaansss/////
Rachael Tatman, J. Vanderplas, and Sohier Dane. A practical taxonomy of reproducibility for machine learning research. 2nd Reproducibility in Machine Learning Workshop at ICML, 6 2018. URL https:
/ //openreview.o wwt.:ee/ddffdddlle/ksss...
Nikolaj Tatti, Taneli Mielikäinen, A. Gionis, and Heikki Mannila. What is the dimension of your binary data? Sizth International Conference on Data Mining (CCDM'06), pp. 603-612, 12 2006. ISSN 1550-4786. doi:
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. ArXiv, abs/22005102243662--66333 2020. URL https://arxi:: org/pdf/ .ttt//2055..00 10243.
Caelano Traina, Agma J. M. Traina, Leejay Wu, and Christos Faloutsos. Fast feature selection using fractal dimension. J. Inf. Data Manag., 1(1):3 16, 2010. ISSN 2178-7107. doi: 10.118///1/6605570x11.
Haruki Tsuchiya, Shinji Fukui, Yuji Iwahori, Yoshitsugu Hayashi, Witsarut Achariyaviriya, and Boonserm Kijsirikul. A method of data augmentation for classifying road damage considering inffuence on classifi- cation accuracy. In Inre J. Rudas, János Csirik, Carlos Toro, János Botzheim, Robert J. Howlett, and Lakhmi C. Jain (eds.), International Conference on Knowledge-Baut Based Intelligent Information & Engineer- ing Systems, volume 159, pp. 1449-1458. Elscvicr BV, 2019. doi: 10.1016/j http://jproos.201111 2019.09.31.
Guillermo Valle-Pérez, Chico Q. Camargo, and Ard A. Louis Deep learning generalizes bocause the parameeter-function map is biased towards simple functions. ICLR, 2019.
Pelar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, P. Lio', and Yoshua Bengio. Graph attention networks. ArXiv, abs/ 11//1171.0.0000 10 2017. URL https://arxiz: wwtp:/t.ggppptpttt /1710. 10903.
Zhi Wan, Yading Xu, and Branko Savija. On the Use of Machine Learning Models for Prediction of Compres- sive Strength of Conerete: Inffuence of Dimensionality Reduction on the Model Performance. Materials, 14(4):713, 2021. ISSN 1996-1944. doi: 00.3390/ma14044p URL https: //wpw mdpi. cwm/ 14/4/713/pd 14/44/712// 211.
34

---

Published in Transactions on Machine Learning Research (10/2024)
Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo. Dkn: Deep knowledge-aware network for news recommendation. In Pierre-Antoine Champin, Fabien L. Gandon, Mounia Lalmas, and Panagiotis G. Ipeirotis (eds.), Proceedings of the 2018 world wide web conference, pp. 1835 1844. ACM, 1 2018. doi: 00.1145/8117886.3188111
Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. Kgat: Knowledge graph atten- tion network for recommendation. Proceedings of the 25th ACM SICKDD International Conference Knowledge Discovery & Data Mining, 5 2019a. doi:
Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. Neural graph collaborative filtering. Proceedings of the 42nd International ACM SIGIR Conference Research and Development in Informa- tion Retrieval, 7 2019b. doi: 10. 10.1145/33331184.33322222
URL https: http://axxvv....g.oo/ ptt/pptt/1905.00100
Xiao Wang, Houye Ji, C. Shi, Bai Wang, Peng Cui, Philip S. Yu, and Yanfang Yc. Hetcrogencous graph atlention network. The World Wide Web Conference, 5 2019c. doi: 10.1145/3005588333336.. URL
https://arxiv. https:://axxiv.rr//dffff /9odf/199...0 111//199933.......
Felix Wu, Tianyi Zhang, Amauri H. de Souza, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. Sim- plifying graph convolutional networks. In PMLR, 2018a.
Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and T. Tan. bession-based recommendation with graph neural networks. ArXio, abs/1811.00885 11 2018b. doi: 10.1609/asa 133i01133301.330 v33101.3301346.
Keyulu Xu, Weihua Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? ArXiu, abs/1810.00882 10 2018. doi:
URL
Liang Yao, Chengsheng Mao, and Yuan Luo. Graph convolutional networks for text classificationn ArXiv, abs/1809.05679, 2018. doi: 10.48550/ 10.48550/arxxv.11600 1rxiv.1809.00679 URL https://arxte http:///xxxvv.oopaaaa////// wtg/p/f//8099 05679.
Rex Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L. Hamilton, and J. Leskovec. Hierarchical graph representation learning with differentiable pooling. Newral Information Processing Systems, pp. 4805-4815, 2018. URL https://export awx.v.. wwwp:ii..rggppf///e/eee 1180/ 08804.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph rastivee learning with augmentations. ArXie, abss/2010.133002:5822 5823, 10 2020. URL https://proceeddng:
Scongjun Yun, Minbyul Jeong, Rachyun Kim, Jacwoo Kang, and Hyunwoo J. Kim. Graph transformer networks. Newral Information Processing Systems, 32:11960- 11970, 11 2019. URL https://pappeet
pdf.
Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5831-5840, 11 2017. doi: 10.1109 20.1099/cppr22018.000
URL sttps://export.art
/1711.
Chuxu Zhang, Dongjin Song, Chao Huang, A. Swami, and N. Chawla. Heterogencous graph neural network. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 0055///222000330000
7 2019. doi:
URL stttps://dl. acm. org/ /tddi//pf///// /10 1144/32925//00 3330961.
Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Neural Information Processing Systems, 31:5171-5181, 12 2018. URL https://arxiv http:/aaxxxvv..rr/eee org/ppf//802..ooo 09691.
Yuhao Zhang, Peng Qi, and Christopher D. Manning. Graph convolution over pruned dependency trees improves relation extraction. ArXiv, Abs/1800.101883 9 2018. doi: 10.18653/v1/dIB 1244. URL https:
//a.xiv.org/pptt//// wtt/pot/t80///999 10185.
Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, L. Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. arXiv: Learning, 6 2020. doi: 10.48550,
1.44500/axxiv.20666 arxiv 11468. URL
wwt///tg////o/oooooooooootoo.tt/ /pt0////0.... 11468.
35

---

Published in Transactions on Machine Learning Research (10/2024)
A Considered Publications
In our survey, we began by collecting a list of publications to consider (Subscction 4.1). We utilized the Semantic Scholar API, employing the search term "graph neural network" to obtain & set of results. These number of citations results were then processed with a dedicated script to calculate the scoring metric 2023-yea of publication for each publication. Out of the vast array of publications, we selected the top 100 papers based on these scores.
The process of reproducing the list proved to be a challenge, because of a significant degree of variability due to the non-deterministic nature of the Semantic Scholar API. This variability was particularly noticeable with regard to page ordering and the contents of the first 10000 entries. Over time we started reproducibility attempts of publications that are now no longer part of the list.
In our filtering process, we manually excluded entries that were surveys, coding frameworks, or lacked a clear connection to graph neural networks. We also ignored works that only applied Graph Convolutional Network (GCN) methods to other field of science or used Lime-dependent or spatial data, especially in the field of chemistry, The reason for this was that the feature selection approach used later was not directly applicable to such work, or at best uninformative. Additionally, publications applying methods to very specific data sets were not included in our list.
Some well-known papers, such as GraphSAINT, were not included because they did not appear under the scarch term used. This absence also explains why SAGN+SLE, SGC, and GraphSAGE do not appear in the list. To better cover the field of graph neural network research, additional search terms like "graph convolutional network" would be necessary. The subsequent changes in criteria led to a high rate of skipped publications in the full list, which was generated end of May 2023.
Table 6: All considered publications with indication on survey status. The indicators are as follows: i included in survey, excluded because experiment is not available, d - excluded because method is only applied on unusual or specially build graph data sets, s skipped because of time constraints.
Publication
Score
Status
Sepe-Supe Classification with GCNs (Kipf & Welling, 2016)
2832.57
Graph Attention Networks (Velickovic et al., 2017)
1869.83
How Powerful are GNNs? (Xu et al., 2018)
913.2
S
Modeling Relational Data with GCNs (Schlichtkrull et al., 2018)
540.0
i
LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation (He
531.0
8
et al., 2020)
Neural Graph Collaborative Filtering (Wang et al., 2019b)
417.0
d
Heterogeneous Graph Attention Network ( (Wang et al,, 2019c)
355.5
S
Hierarchical Graph Representation tion Learning with Differentiable Pooling (Ying et al., 2018)
322.4
i
Graph Contrastive Learning with Augmentations (You et al., 2020)
303.67
8
KGAT: Knowledge Graph Attention Network for Recommendation (Wang et al., 2019a)
281.5
8
GCNs for Text Classification (Yao et al., 2018)
261.4
d
Link Prediction Based on GNNs (Zhang Chen, 2018)
248.4 4
s
GCNs for Hyperspectral Image Classification (Hong et al., 2020)
234.0
d
DeepGCNs: Can GCNs Go As Deep As CNNs? (Li et al., 2019)
230.75
S
E(n) Equivariant GNNs (Satorras el al., 2021)
220.5
Weisfeiler and Leman Go Neural: Higher-order GNNs (Morris et al., 2018)
211.0
Predict then Propagate: GNNs meet Personalized PageRank (Kliepera et al., 2018)
210.6
8
Heterogeneous Graph Transformer (Hu et al., 2020b)
209.0
8
Heterogeneous GNN (Zhang et al., 2019)
203.0
8
Geom-GCN: Geometric GCNs (Pei et al., 2020)
196.67
8
Session-basod Recommendation with GNNs (Wu et al., 2018b)
193.0
Self-Attention Graph Pooling (Lee et al., 2019)
180.5
GCC: Graph Contrastive Coding for GNN Pre- Traininn (Qiu et al., 2020)
178.0
8
Few-Shot Learning with GNNs (Satorras & Bruna, 2017)
169.33
8
Dynamic Edge- Conditioned Filters in CNNs on Graphis (Simonovsky & Komodakis, 2017)
168.67
36

---

Published in Transactions on Machine Learning Research (10/2024)
How to Find Your Friendly Neighborhood:
Graph Attention
Design
with
Self-
159.0
S
Supervision (Kim & Oh, 2022)
Beyond Homophily in GNNs: Current Limitations and Effective Designs (Zhu et al., 2020)
158.33
S
GNN-Based Anomaly Detection in Mullivariate Time Series (Deng Hooi, 2021)
150.0
DKN: Deep Kuowledge-Award Aware Network for News Recommendation (Wang et al., 2018)
149.8
d
MixHop: Higher-Ordes Graph Convolutional Architectures via Sparsified Neighborhood Mix-
142.0
8
ing Abuu-llHaija al., 2019)
MAGNN: Metapath Aggregated GNN for Heterogeneous Graph Embedding (Fu et al., 2020)
137.67
Principal Neighbourhood Aggregation for Graph Nets (Corso et al., 2020)
132.67
8
Hypergraph Neural Networks (Feng al., 2018)
128.6
Graph Transformer Networks (Yun et al., 2019)
128.25
Beyond Low-frequency Information in GCNs (Bo et al., 2021)
125.0
8
Encoding Sentences with GCNs for Semantic Role Labeling (Marcheggiani & Titov, 2017)
123. 17
Neural Motifs: Scene Graph Parsing with Global Context (Zellers et al., 2017)
122.67
d
Graph Convolution over Prumed Dependency Trees Improves Relation Extraction (Zhang
122.4
et al., 2018)
Graph Structure Learning for Robust GNNs (Jin et al., 2020)
121.33
8
Towards Deeper GNNs (Liu et al., 2020b)
118.67
8
B Reproducibility Context
The data collected during the successful reproducibility attempts in Section 4 can be summarized in a Formal Context (Ganter & Wille, 1997), where the papers make up the object set and the analyzed features of reproducibility the attribute set. A cross for paper p and feature f mcans that this feature was observed for the paper. Based on the definitions of the features, this indicates negative aspect of reproducibility occurring in the paper.
37

---

Published in Transactions on Machine Learning Research (10/2024)
Table 7: Formal Context derived from Reproducibility Survey with adjoining error ontology.
The table is rotated sideways, e.g. Papers are objects and reproducibility features are attributes.
R6 no prodictions
Sompyraee Ponne ivsilt
R5 weak statisties
predictions
R4 strong differences everywhere
R3 strong differences parts
x
small deviations
X
model
RI model weights
X
only general. implemented
S18
hyperparamneter search not included
experiments
S17-au all missing
S16 missing
code
our memorry
S14 api changes
bugs
813 fix distributed thzough other channels
812
issue solutions not applied
never
unn-on which
scripts
89 incomplete train /tost scripts
ss missing hyperparameters
docamentatioon
S7 noresarry arguments not clear
S6 date
importanc
variables
$4 sceds not set
83 uacesssry hardwaee musviiabble
dependencies
82 speeiflfee version not available anymore
S1 exact version not documented
cum-err of samples not documented
D10 D10-traln/see /tott splits uncleur
trausformation
D9 incomplete deseription
preproccessing
De manmal steps
D7 request only
set
dowalood
0i-piivacy restricted
asailability
D4-acose not possible
D no direct acces
D2 version not specified
motadata
D1 format not documented
GON Releiionl GCN
GraphSSnge
Difgooo SGC SAGIN-SLEE
38

---

Published in Transactions on Machine Learning Research (10/2024)
C Presentation of other Experiments
Here we want include figures presenting the results from the influence experiments not yet presented in detail. The diagrams structured the same way the ones presented in Subsection 5.4 For cach experiment first show the normalized distribution of normalized intrinsic dimensionality for the used data sets (after preprocessing). For data sets with more than 10 samples, an algorithm for calculating the approximated NID is used. Additionally a second figure presents the accuracy /fl scores obtained when training on the feature reduced data sets. For a more detailed explanation see description accompanying Figure 3.
Some data sets have few features that the steps of the discarding process are smaller than one feature. This leads to fewer data points, which in turn give the impression of only partially complete graphs for visualizations of normalized distribution of the NID or accuracy for given discarding proportions the necessary gramularity can not be achieved (see Figure 6).
munce
a/D), max, a(D)
Accuracy
0.6
0.5
0.8
1.0
122
0.5
0.6
0.9
Feature (relative ranked position)
Discarded Proportion a
(a) Normalized intrinsic dimensionality.
(b) Performance of resulting model.
Figure 6: Experiment based R-GCN.
The DiffPool publication used only two data sets, of which only one, namely the enzymes data set, has features for the graph nodes. Therefore we were able apply the described method to the other data encountered another problem during the DiffPool experiments (see Figure 7). It is
set. Furthermore strongly implied in the paper that the method uses the node features in its computations. However, close cxamination of the source code reveals that in the default configuration, the node fcatures are built from the classification targets of the associated graph. By changing the corresponding parameler in the training script to a different argument, which we decided on the basis of which preprocessing modifications induced, no improvements could be observed. On the contrary, the overall performance of the method got worse. Nevertheless, we present the results obtained, which again show, that the DiffPool method does not use the node features in comprehensible way.
39

---

Published in Transactions on Machine Learning Research (10/2024)
10a
0.425
a(D) ( maxy a(D)s
10
Accuracy
0:575
0.350
10
0.325
10-3
0:275
0.4
0.8
1.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Feature (relative ranked position)
Discarded Proportion
(a) Normalized intrinsic dimensionality. Figure 7: Experiment based on DiffPool.
(b) Performance of resulting model.
10a
roddis
a(D), maxy' 0(D),
0.8
score
(stegg)
10
(eg)
(suapp)
f1
(sug)
0.6
10:2
0.4
1.0
0.0
0.2
0.3
0.5
0.6
0.8
0.9
Feature (relative ranked position)
Discarded Proportion a
(a) (Approximated) normalized intrinsic dimensionality.
(b) Performance of resulting model. Figure 8: Experiment based on GraphSAGE.
0.050
10-1
Fodi1
0.80
00.445
8/D)// maxy )(D)
10-2
0.78
0.948
Accuracy
5core
0..444
0.76
cors
00.542
10-4
-0.940
10
0.72
0.908
10-6
0.908
0.0
0.2
0.4
0.6
0.8
1.0
0.1
3.4
01.
Feature (relative ranked position)
Discarded Proportion a
(a) (Approximated) normalized intrinsic dimensionality.
(b) Performance of resulting model.
Figure 9: Experiment based on SGC.
40


Reviews:
[{"review_id": "ATszSYq3xi", "reviewer": "Reviewer_qkD2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe paper proposes a collection of requirements to judge whether an ML paper is reproducible. This contribution is a valuable attempt for the community, which still lacks a shared set of guidelines to evaluate the reproducibility of ML studies. \n\n\n**Weaknesses**\n\nThe first part of the paper, which concerns reproducibility, and the second, which focuses on feature selection, have very little in common to the point that they could be regarded as two standalone papers.  \\\nWhen the data ID is high, a ML algorithm might generalize poorly out-of-sample, i.e., the test error might be higher, but a high test error does not make a ML study less reproducible. Indeed, the value of the test error can be perfectly reproduced. \n\n> Q1: *What is the connection between a machine-learning paper's reproducibility and the data's intrinsic dimension?  Are, for instance, results obtained from high dimensional data systematically less/more reproducible than those obtained from low dimensional data?*\n\nThe authors claim that Section 5's main goal is to “investigate the influence of intrinsic dimensionality on model behavior.” However, Section 5 discusses the impact of feature selection on model performance. The ID is never directly linked to the model performance, and its value is never shown: in Figs 3a and 4a, the ID is normalized so that its value always goes from 0 to 1. \n\n> Q2: *Is there any evidence in the experiments shown in the paper that the data intrinsic dimension correlates with model performance?*\n\nOn a high level, the section shows that feature selection can be applied without decreasing accuracy. For example, if one keeps 40% of the features (Fig 3/4b and Fig 5), the accuracy is not compromised. This is a well-known fact (see, e.g., https://link.springer.com/article/10.1007/s10462-019-09682-y). \n\n> Q3 *What kind of new knowledge or perspective does the proposed ID-based feature selection bring us?*", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The contribution of the paper is twofold. \n\nFirst, it presents an ontology to determine if a machine-learning paper is reproducible and applies the proposed ontology to analyze the reproducibility of papers in the field of graph convolutional networks. \\\nSecond, it analyzes the effect of a concentration-based estimate of the intrinsic dimension (ID) on the model performance. In this second part, the authors mainly focus on the impact of an ID-based feature selection on model accuracy.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. A convincing answer to questions Q1 and Q2 in the weaknesses section is critical for acceptance. \n\n2. In particular, a quantitative connection between the dataset's intrinsic dimension (not just normalized by the maximum dataset value) and reproducibility, or at least its impact on model performance, is critical for acceptance. \n\n3. How do the results shown in section 5.3 depend on the specific choice of feature functions? The authors in the paper discuss the case of $f_j(x) = x_j$. It would be nice to show that the results do not substantially depend on this choice.", "broader_impact_concerns": "No ethical implications or concerns would require adding a broader impact for the current work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SimZfX16WW", "reviewer": "Reviewer_1k1o", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "A clear strength of this work is the rigorous ontology (and corresponding pipeline) that has been proposed. It is complete and exhaustive, and, I believe, very useful for all machine learning practitioners who care about reproducibility. Its application to graph neural network projects is also quite interesting, and allows a clear high level picture of the different approaches and how easy it is to re-use (some of) them. Hence, I think this article provides a good contribution to the machine learning field.\n\nI do have a few concerns though:\n\n---The writing could be greatly improved. I had a lot of trouble understanding some paragraphs (see also below), and there are numerous confusing typos in the text.\n\n---The investigation of intrinsic dimension is not so compelling. I think that the intrinsic dimension has been investigated under various other lenses (for instance, see https://proceedings.neurips.cc/paper/2020/file/54f3bc04830d762a3b56a789b6ff62df-Paper.pdf), and that a proper comparison (or at least some discussion in the text) should be implemented in order to better highlight the specificity of the dimension used in this work. I also find it a bit difficult to grasp the message conveyed by this last section: how should I interpret that GCN is more robust than GraphSAGE for instance? Should I understand that I can safely discard the high-NID features when using GCN models (and thus save on running time and memory), while it is dangerous for GraphSAGE? This should appear more explicitly.\n\n---Related to this comment, I think the proposed ontology could be improved by also designing, e.g., some sort of reproducibility score, in order to formally compare and rank papers. Otherwise, it is a bit unclear how the question tree derived from the ontology can help in trimming the papers from the literature.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this article, the authors propose a new ontology, taking the form of a set of guidelines organized into questions, for assessing and evaluating the reproducibility of machine learning projects. This ontology is very general (i.e., not specific to particular fields of data science), and covers an exhaustive list of aspects related to experimental claims and evidence, including (but not limited to) availability of source code, data sets and documentation. The authors then applied their ontology to graph neural networks, and, among the papers that were deemed reproducible enough, investigated the influence on the intrinsic dimension of data sets on the different model performances. By characterizing how data set features concentrate (i.e., aggregate towards their mean or median), and removing them progressively, they were able to detect those models that are very sensitive to these removals (e.g., GraphSAGE), and those that are more robust (e.g., Graph Convolutional Networks).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "---There a few sentences/paragraphs that I could not understand. Could the authors rephrase or explain in greater details?\np11: \"Furthermore, the collection [...]\"\np14: \"Such a function allows for estimating [...]\"\np17: \"For this, we first extract the logic [...]\"\n\n---p9, criterion R5: setting a threshold at 5 samples sounds arbitrary. Is there some justification? Also, I think this is not very relevant for some machine learning fields such as one-shot learning, where learning data is very restricted. \n\n---p15: I do not think it is worth introducing Borel measures in Definition 1, as they are not used later.\n\n---p15: d_F is only a pseudo-metric, not a proper metric.\n\n---Here is a (non exhaustive) list of typos:\np1: \"isto\"\np3: \"with in influence\"\np3: \"those those\"\np5: \"now\" --> new\np7: \"sideeffects\"\np7: \"me\" --> be\np8: \"outline\" --> outlined\np10: \"We starting\"", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DOmhOHtal4", "reviewer": "Reviewer_gYjg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1: The paper effectively contextualizes the research within the broader landscape of reproducibility and dimensionality reduction in machine learning, demonstrating a strong theoretical foundation for the study.\n\n2: The systematic approach to studying reproducibility in graph neural network research, including the selection of papers and the development of a scoring metric, showcases methodological rigor and attention to detail.\n\n3: The investigation into the influence of intrinsic dimension on ML training processes and the manipulation of intrinsic dimension in graph data sets provides valuable insights for researchers and practitioners in the field.\n\n4: By incorporating V. Pestov's geometric approach for estimating intrinsic dimension, the paper offers a novel perspective on addressing reproducibility challenges in high-dimensional ML settings.\n\nWeaknesses:\n1: While the paper presents a theoretical framework for studying reproducibility and intrinsic dimensionality, the absence of empirical validation or case studies limits the practical applicability of the findings.\n\n2: The paper briefly touches on the challenges of implementing exact experimental procedures but lacks a detailed discussion on specific implementation hurdles faced by researchers in reproducing graph neural network results.\n\n3: It would be better if the authors can include discussions of more representative GNN approaches, such as the following ones:\nHOW POWERFUL ARE GRAPH NEURAL NETWORKS? ICLR'19\nLearning to drop: Robust graph neural network via topological denoising. WSDM'20.\nRobust Graph Representation Learning via Neural Sparsification. ICML'19\n\n4: Some sections of the paper could benefit from improved clarity and organization to enhance readability and facilitate understanding for a broader audience. Basically, the current writing is hard to follow.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The study explores reproducibility challenges in machine learning, particularly in graph neural networks, and investigates the impact of intrinsic dimensionality on model training processes. By analyzing the methodology, results, and implications of the research, this review aims to highlight the paper's contributions and areas for improvement.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1). Better writing to make the paper easy to follow. Possibly, tempirical validation or case studies would be helpful.\n\n2). more discussion of representative GNN models.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a rigorous ontology to test reproducibility of papers in ML, and will thus potentially be a meaningful reading for all researchers processing large data bases. The proposed method is tested on multiple papers with success. The reviewers all agree of the interest to the TMLR community and beyond. However, there is also agreement on the fact that the paper would benefit greatly of some improvement in the writing. I this recommend acceptance, conditional on the authors taking this last comment into account, who would thus publish a final version with an improved presentation.
Decision: Accept (poster)