## Abstract

**Summary:** PAX2GRAPHML is an open-source Python library that allows to easily manipulate BioPAX source files as regulated reaction graphs described in graphml format. The concept of regulated reactions, which allows connecting regulatory, signaling and metabolic levels, has been used. Biochemical reactions and regulatory interactions are homogeneously described by regulated reactions involving substrates, products, activators and inhibitors as elements. PAX2GRAPHML is highly flexible and allows generating graphs of regulated reactions from a single BioPAX source or by combining and filtering BioPAX sources. Supported by the graph exchange format .graphml, the large-scale graphs produced from one or more data sources can be further analyzed with PAX2GRAPHML or standard Python and R graph libraries.

**Availability and implementation:** https://pax2graphml.genouest.org.

**Contact:** francois.moreews@irisa.fr

## 1 Introduction

BioPAX is a standard format encoding biological processes like gene regulation, metabolic pathways or signaling events, that facilitates the inter-operability between data sources and network analysis tools. However, this rich knowledge-oriented data format that finely captures the complexity of biological networks cannot be easily handled without appropriated tools. Software have been recently proposed to design, visualize (Babur et al., 2010; Shannon et al., 2003), parse (Turei et al., 2016), validate (Rodchenkov et al., 2013), query (Babur et al., 2014) and analyze BioPAX files. However, an important missing feature to analyze BioPAX data sources is the ability to interpret BioPAX files into graph structures including the role of physical entities as substrate, product or regulator in the reactions.

An accurate format for representing the variety and complexity of the biological reactions is the concept of regulated reactions connecting regulatory, signaling and metabolic levels (Blavy et al., 2014). In this conceptual framework, both biochemical reactions and regulatory interactions are described homogeneously as regulated reactions involving substrates, products, activators, inhibitors and modulators as key elements. In the reaction graph generated from regulated reactions, the molecules and the reactions are represented as typed nodes, as shown in Figure 1.

Thus, we propose to extend the BioPAX toolbox with a Python library able to interpret BioPAX files as graphs of regulated reactions. With PAX2GRAPHML, the graphs are represented in the .graphml format, allowing the manipulation of nodes and edges

<figure><img src="image_58.png" /><figcaption>Fig. 1. Example of reaction graph manipulated by PAX2GRAPHML showing reactions and entities as nodes</figcaption></figure>

properties. The PAX2GRAPHML tool also enables extracting subgraphs, by filtering the original files according to specific properties of the nodes (genes or proteins) or by merging different graphs. It also implements basic methods to explore the graphs. Thanks to the .graphml exchange format support, generated graphs can be further analyzed with already existing graph libraries in Python or R.

## 2 Format and package description

PAX2GRAPHML is able to process all BioPAX files to generate regulated reaction graphs, which can be further interpreted into positive and negative oriented influences. It is available on pypi and as a docker image.
<table><caption>**Table 1.** BioPAX files transformation of datasources available in PC into regulated reaction graphs with PAX2GRAPHML</caption><thead><tr><th>Data sources</th><th>Nodes</th><th>Reaction nodes</th><th>Entity nodes</th><th>Edges</th><th>Substrate of</th><th>Product of</th><th>Activator of</th><th>Inhibitor of</th></tr></thead><tbody><tr><td>PC* all sources</td><td>175 262</td><td>85 750</td><td>89 512</td><td>639 945</td><td>84 496</td><td>95 743</td><td>52 009</td><td>407 697</td></tr><tr><td>CTD</td><td>44 639</td><td>19 814</td><td>24 825</td><td>98 993</td><td>18 538</td><td>19 073</td><td>35 077</td><td>26 305</td></tr><tr><td>HumanCyc</td><td>5733</td><td>1778</td><td>3955</td><td>11 890</td><td>3875</td><td>4455</td><td>3560</td><td>0</td></tr><tr><td>INOH</td><td>4315</td><td>2188</td><td>2127</td><td>7409</td><td>4247</td><td>3162</td><td>0</td><td>0</td></tr><tr><td>Intact complex</td><td>2187</td><td>563</td><td>1624</td><td>2869</td><td>2306</td><td>563</td><td>0</td><td>0</td></tr><tr><td>KEGG**</td><td>3133</td><td>1560</td><td>1573</td><td>7041</td><td>3488</td><td>3553</td><td>0</td><td>0</td></tr><tr><td>Mirtarbase</td><td>32 727</td><td>15 064</td><td>17 663</td><td>395 703</td><td>0</td><td>15 064</td><td>0</td><td>380 639</td></tr><tr><td>Panther</td><td>3766</td><td>1662</td><td>2104</td><td>5263</td><td>2914</td><td>2165</td><td>142</td><td>42</td></tr><tr><td>PID</td><td>9403</td><td>4495</td><td>4908</td><td>14 827</td><td>6613</td><td>4544</td><td>3233</td><td>437</td></tr><tr><td>Reactome</td><td>31 718</td><td>11 404</td><td>20 314</td><td>46 541</td><td>27 210</td><td>15 358</td><td>3717</td><td>256</td></tr><tr><td>Reconx</td><td>6956</td><td>2821</td><td>4135</td><td>20 485</td><td>7722</td><td>7689</td><td>5074</td><td>0</td></tr><tr><td>PC* all sources except CTD and Mirtarbase</td><td>119 285</td><td>55 184</td><td>64 101</td><td>167 515</td><td>83 782</td><td>66 056</td><td>16 939</td><td>7 38</td></tr><tr><td>PID and HumanCyc</td><td>12 561</td><td>4518</td><td>8043</td><td>16 085</td><td>3575</td><td>5859</td><td>6221</td><td>430</td></tr><tr><td>PID and HumanCyc and KEGG</td><td>15 564</td><td>6079</td><td>9485</td><td>19 651</td><td>5332</td><td>7652</td><td>6237</td><td>430</td></tr><tr><td>PID and HumanCyc and KEGG and Reactome</td><td>38 585</td><td>10 398</td><td>28 187</td><td>43 899</td><td>16 087</td><td>17 371</td><td>9756</td><td>685</td></tr></tbody></table>

Note: Single datasources transformations were performed with the sub-package *pax_import*. Combination of several datasources was performed by filtering the PC* all sources graphml file with the sub-package extract. Nodes are either reactions or entities (proteins, small molecules, etc.). The numbers of regulated reactions computed by PAX2GRAPHML, together with the number of substrates, products, activators, inhibitors, are indicated. All these graphs can be directly downloaded from the PAX2GRAPHML website.

**PC* version 12, September 2019.

**KEGG, July 2011 (only human, hsa* files).

In PAX2GRAPHML, PaxTools (Babur et al., 2014) is used internally to extract sub-classes of patterns and further interpret them as regulated reactions. These extracted patterns form the building elements of a *regulated reaction graph* (Blavy et al., 2014). Each regulated reaction graph pattern is centered on a reaction node linked to one or several substrate nodes and product nodes. The reaction node can also be linked to modulator nodes (activators or inhibitors). Substrates and modulators are inputs of the reaction node, whereas products are outputs of the reaction node. All nodes (reaction, substrate, product or modulator) are associated with their own metadata in the graph.

PAX2GRAPHML is composed of four sub-packages. (i) The sub-package *pax_import* is dedicated to global or parametrized import of BioPAX files from Pathway Commons (PC) to be further interpreted as regulated reaction graph. (ii) The sub-package properties allow to manipulate nodes and edges properties of the generated graphs. All aliases contained in BioPAX have been incorporated in the .graphml format as node properties to represent genes, protein and compounds. Additional annotations can also be directly imported from specific files. (iii) The sub-package extract allows modifying either the generated reaction graph or the influence graph, including sub-graphs selection or graphs merging. (iv) The sub-package *graph_explore* includes IO functions and analysis of the generated graphs. It also includes classical graph metrics (degree, betweenness, closeness, connected components) as preliminary steps. More sophisticated analyses can be further performed with graph-tool or other advanced libraries (Csardi and Nepusz, 2006).

The PAX2GRAPHML website provides a complete documentation and the pre-processed database resources. Regulated reaction graphs and influence graphs produced from 16 data sources of PC can be downloaded as ready-to-use data for further analyses with PAX2GRAPHML. Files are automatically updated using databanks synchronization and a processing software (Filangi et al., 2008).

## 3 Application

PAX2GRAPHML was first applied to the complete PC databank. The regulated reaction graph produced in.graphml format has a size of 363 MB (13% of the initial BioPAX file size). PAX2GRAPHML

was also applied to each data source of PC considered independently. As shown in Table 1, the regulated reaction concept used to unify the different BioPAX reaction types facilitates the comparison of the content of each resource. Notably, this revealed that Mirtarbase and CTD are the main contributors of PC in terms of nodes, edges, and especially inhibition reactions.

Generating the regulated reaction graph from 16 BioPAX datasources with PAX2GRAPHML lasted 7 days on a virtual machine with 48 G RAM. Conveniently, the generated files can be downloaded on PAX2GRAPHML website as ready-to-use data resources, which is automatically updated.

Customized graphs can be produced for any subsets of the databases. To achieve this, users can either filter the overall regulated reaction graph, or can merge the regulated reaction graphs produced from two or more databases selected according to their specific interest. The two functionalities (filtering and merging) are available within the PAX2GRAPHML package. As an illustration, Table 1 shows that filtering out CTD and Mirtarbase from PC eliminates 32% of the nodes (36% of reaction nodes and 28% of entity nodes) and 74% of the edges. Table 1 also illustrates that the combination of PID with successively HumanCyc, KEGG and Reactome improves coverage of both reaction nodes (from 4495 to 10 398) and entities (from 4908 to 28 187).

By managing BioPAX data extraction into regulated graphs, PAX2GRAPHML simplifies the implementation of many methods for regulation network analysis and understanding of the controlling steps of the biological pathways.

*Financial Support:* none declared.

*Conflict of Interest:* none declared.

## References

* Babur, O. et al. (2010) ChiBE: interactive visualization and manipulation of BioPAX pathway models. *Bioinformatics*, 26, 429–431.

* Babur, O. et al. (2014) Pattern search in BioPAXmodels. *Bioinformatics*, 30, 139–140.
Blavy,P. et al. (2014) Using a large-scale knowledge database on reactions and regulations to propose key upstream regulators of various sets of molecules participating in cell metabolism. *BMC Syst. Biol.*, **8**, 32.

Csardi,G. and Nepusz,T. (2006) The igraph software package for complex network research. *InterJ. Complex Syst.*, **1695**, 1–9.

Filangi,O. et al. (2008) BioMAJ: a flexible framework for databanks synchronization and processing. *Bioinformatics* (Oxford, England), **24**, 1823–1825.

Rodchenkov,I. et al. (2013) The BioPAX validator. *Bioinformatics*, **29**, 2659–2660.

Shannon,P. et al. (2003) Cytoscape: a software environment for integrated models of biomolecular interaction networks. *Genome Res.*, **13**, 2498–2504.

Turei,D. et al. (2016) OmniPath: guidelines and gateway for literature-curated signaling pathway resources. *Nat. Methods*, **13**, 966–967.
<figure><img src="image_59.png" /></figure>

HAL
open science

# Towards a reproducible interactome: semantic-based detection of redundancies to unify protein-protein interaction databases

Marc Melkonian, Camille Juigné, Olivier Dameron, Gwenaël Rabut, Emmanuelle Becker

<figure><img src="image_60.png" /></figure>

## To cite this version:

Marc Melkonian, Camille Juigné, Olivier Dameron, Gwenaël Rabut, Emmanuelle Becker. Towards a reproducible interactome: semantic-based detection of redundancies to unify protein-protein interaction databases. Bioinformatics, Oxford University Press (OUP), 2022, pp.1-7. 10.1093/bioinformatics/btac013. hal-03522989

HAL Id: hal-03522989

https://hal.archives-ouvertes.fr/hal-03522989

Submitted on 12 Jan 2022

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.
# Towards a reproducible interactome: semantic-based detection of redundancies to unify protein-protein interaction databases

Marc Melkonian, Camille Juigné, Olivier Dameron, Gwenaël Rabut, Emmanuelle Becker

► To cite this version:

Marc Melkonian, Camille Juigné, Olivier Dameron, Gwenaël Rabut, Emmanuelle Becker. Towards a reproducible interactome: semantic-based detection of redundancies to unify protein-protein interaction databases. Bioinformatics, Oxford University Press (OUP), 2022, 10.1093/bioinformatics/btac013. hal-03522989

HAL Id: hal-03522989

https://hal.archives-ouvertes.fr/hal-03522989

Submitted on 12 Jan 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
# Towards a reproducible interactome: semantic-based detection of redundancies to unify protein-protein interaction databases

Marc Melkonian<fnref n="1" />,<fnref n="2" />, Camille Juigné<fnref n="1" />,<fnref n="3" />, Olivier Dameron<fnref n="1" />, Gwenaël Rabut<fnref n="2" />,*
and Emmanuelle Becker<fnref n="1" />,*

Associate Editor: XXXXXXX

Received on XXXXX; revised on XXXXX; accepted on XXXXX

## Abstract

**Motivation:** Information on protein-protein interactions is collected in numerous primary databases with their own curation process. Several meta-databases aggregate primary databases to provide more exhaustive datasets. In addition to exhaustivity, aggregation contributes to reliability by providing an overview of the various studies and detection methods supporting an interaction. However, interactions listed in different primary databases are partly redundant because some publications reporting protein-protein interactions have been curated by multiple primary databases. Mere aggregation can thus introduce a bias if these redundancies are not identified and eliminated. To overcome this bias, meta-databases rely on the Molecular Interaction ontology that describes interaction detection methods, but they do not fully take advantage of the ontology's rich semantics, which leads to systematically overestimating interaction reproducibility.

**Results:** We propose a precise definition of explicit and implicit redundancy, and show that both can be easily detected using Semantic Web technologies. We apply this process to a dataset from the APID meta-database and show that while explicit redundancies were detected by the APID aggregation process, about 15% of APID entries are implicitly redundant and should not be taken into account when presenting confidence-related metrics. More than 90% of implicit redundancies result from the aggregation of distinct primary databases, while the remaining occurs between entries of a single database. Finally, we build a "reproducible interactome" with interactions that have been reproduced by multiple methods or publications. The size of the reproducible interactome is drastically impacted by removing redundancies for both yeast (-59%) and human (-56%), and we show that this is largely due to implicit redundancies.

**Availability:** Software, data and results are available at https://gitlab.com/nnet56/reproducible-interactome, https://reproducible-interactome.genouest.org/, Zenodo (doi:10.5281/zenodo.5595037) and NDEx (doi:10.18119/N94302, doi:10.18119/N97S4D

**Contact:** emmanuelle.becker@irisa.fr, gwenael.rabut@inserm.fr

**Supplementary information:** Supplementary data are available at *Bioinformatics* online.

## 1 Introduction

Protein-protein interactions (PPIs) play an ubiquitous and fundamental role in all biological processes. Description of PPIs is essential to understand how proteins operate at the molecular level and the construction of accurate

<figure><img src="image_61.png" /></figure>

<figure><img src="image_62.png" /></figure>

<figure><img src="image_63.png" /></figure>

<figure><img src="image_64.png" /></figure>

<footnote n="1">¹Univ Rennes, Inria, CNRS, IRISA, F-35000, Rennes, France</footnote>

<footnote n="2">²Univ Rennes, CNRS, IGDR - UMR 6290, F-35000, Rennes, France</footnote>

<footnote n="3">³Pegase, Inrae, Institut Agro, 35590 Saint-Gilles, France.</footnote>

<footnote>* To whom correspondence should be addressed, equal contribution.</footnote>
and comprehensive protein interaction networks (or interactomes) is an important aim of biological research (Bonetta, 2010; Cafarelli et al., 2017; Luck et al., 2020; Huttlin et al., 2021).

PPIs can be probed using numerous interaction detection methods (IDMs), following biophysical (e.g. x-ray crystallography), biochemical (e.g. affinity purification) or genetic approaches (e.g. yeast two-hybrid). Importantly, since different IDMs probe PPIs in a different manner, they produce complementary results that often do not fully overlap. For instance, some IDMs are designed to detect binary interactions of proteins probed in pairs (e.g. yeast two-hybrid), while others probe interactions of protein groups assembled in complexes (e.g. affinity purification). Consequently, the biological interpretation of PPI networks depends on the underlying IDMs that have been used to produce them. Moreover, since IDMs can generate false positive and false negative interactions, multiple observations of a given PPI with different experimental techniques reinforce the confidence in this PPI. Accurate IDM annotation and interpretation is thus an important issue in interactome studies.

Information on published PPIs is collected in primary databases such as IntAct (Kerrien et al., 2012), MINT (Calderone et al., 2020), BioGRID (Oughtred et al., 2019), DIP (Salwinski et al., 2004) or HPRD (Keshava Prasad et al., 2009). The major databases report IDMs using a controlled vocabulary defined by the Proteomics Standards Initiative-Molecular Interactions (PSI-MI) consortium (Sivade Dumousseau et al., 2018). This vocabulary is structured in an ontology to represent the hierarchical relationships between IDM families by a directed acyclic graph.

Each primary database follows its own curation process with different literature mining, filtering, and reporting techniques. To address the resulting need for integration, several meta-databases aggregate information from multiple primary databases to provide more exhaustive PPI datasets. Some of these meta-databases, such as the Agile Protein Interactomes DataServer (APID) (Alonso-López et al., 2016; Alonso-López et al., 2019), HINT (Das and Yu, 2012) or mentha (Calderone et al., 2013), focus exclusively on experimentally determined PPIs, while others, such as IID (Kotlyar et al., 2019) or STRING (Szklarczyk et al., 2019) also integrate predicted interactions, text mining results or other information.

The accurate aggregation of PPIs from multiple and partly redundant sources is not a trivial task (Turinsky et al., 2010; Klapa et al., 2013). Although the primary databases refer to the PSI-MI ontology, they do not necessarily select identical terms to annotate PPIs (Alonso-López et al., 2019). Hence, a PPI observed in a single experiment reported in a given publication can be annotated with distinct IDM terms in different primary databases. Such annotation differences are usually not taken into account or corrected during the aggregation process.

APID, which unifies data from five of the largest PPI databases (Alonso-López et al., 2016; Alonso-López et al., 2019), implements an integration method that takes redundancy into account and enables to distinguish 'experimental evidences' (i.e experimental observations reported in publications) from 'curation events' (i.e. entries in PPI databases). For a given protein pair, multiple entries annotated with identical IDM and identical PubMed publication identifier (PMID) are considered as duplicates and counted as a single experimental evidence. In addition, IDMs are classified into 'binary' and 'indirect' methods and IDMs corresponding to related binary methods (e.g. 'two hybrid array' and 'two hybrid pooling approach') are assigned a common method type (e.g. 'two hybrid'). This common method type is then used instead of the original IDM to identify duplicate entries across multiple databases. This custom integration process is not fully satisfying since it is restricted to binary interactions and it does not take advantage of the PSI-MI ontology.

We propose a novel approach to integrate PPI information from primary databases. We define the conventional explicit redundancy and extend it with implicit redundancy based on parent-related terms in the PSI-MI

ontology. We present a method relying on Semantic Web technologies that successfully detects and reconciles implicit redundancies in curation events compiled from multiple primary databases, opening the way to an improved automated curation process. Once curated for both explicit and implicit redundancies, the integrated set of experimental evidences can be used to determine the reproducible interactome supported by multiple experiments.

## 2 Approach

### 2.1 Explicit and implicit redundancy

Let us consider a pair of proteins (A, B) and count the number of non-redundant experiments reporting their interaction.

Primary databases such as BioGRID or IntAct can provide several entries corresponding to this protein pair. Usually, these entries differ in the IDM, the PMID, or both. An entry in these databases can thus be defined by a quadruplet

$$ (A, B, M_i, P_x) $$

where A and B are the proteins, $M_i$ is the IDM (such as 'affinity chromatography technology', 'anti-tag coimmunoprecipitation' or 'two hybrid', for the most frequent ones), and $P_x$ is the PMID of the original article describing their interaction. When two entries only differ in the IDM, this should signify that the original article has observed the interaction using several experimental techniques. When two entries only differ in the PMID, this should signify that the interaction has been reproduced in two distinct studies using the same detection method.

For meta-databases such as APID, populated by aggregating curation events from other databases, an entry can be defined by a quintuplet

$$ (A, B, M_i, P_x, D_a) $$

where $D_a$ indicates the primary database indexing the interaction. Meta-databases can contain different types of redundancies:

* **Explicit redundancy** occurs when distinct entries referring to the same protein pair (A, B) and the same PMID $P_x$ have an identical IDM $M_i$. This happens when two primary databases registered the same experimental evidence using the same IDM term. Explicit redundancies are detected and unified by APID and other meta-databases.

* **Implicit redundancy** occurs when distinct entries referring to the same protein pair and the same PMID have been annotated with different IDMs although they correspond to the same experimental evidence. In practice, this occurs when curators select IDM terms at different levels of the ontology, one being more general and the other more specific. For example, the interaction of the human proteins MDM2 and TP53 is listed in APID as (MDM2, TP53, 'anti tag Co-immunoprecipitation', PMID:17159902, INTACT:7156209) and also as (MDM2, TP53, 'affinity chromatography technology', PMID:17159902, BIOGRID:680279). Although biologists would naturally recognize one observation annotated twice at different granularities, the redundancy is not explicit. Implicit redundancy should not be confused with the common case where several experimental techniques are used in a single publication to validate a given PPI. Therefore, detecting implicit redundancies requires knowledge on IDMs.

Hereafter, we take advantage of the PSI-MI ontology to identify these two cases, as illustrated in Figure 1.
<figure><img src="image_65.png" /><figcaption>Fig. 1. Illustration of the different types of redundancy across primary databases. (A) Curation events from two databases (DB_1 and DB_2). Depending on the IDM reported by DB_2, one can identify no redundancy (purple), explicit redundancy (yellow), or implicit redundancy (blue). Ontology representations of the different cases are presented in panels (B), (C) and (D).</figcaption></figure>

## 2.2 Definitions

Following the notation introduced in 2.1, we consider two entries, $E_i$ and $E_j$, of a meta-database, defined by their respective quintuplets of the form $(A, B, M_i, P_x, D_\alpha)$. Note that here we do not consider the experimental role of A and B, therefore all PPIs are symmetric and the order of A and B is irrelevant.

$E_i$ and $E_j$ present explicit redundancy if and only if:

$$ \begin{cases} E_i = (A, B, M_i, P_x, D_\alpha) \\ E_j = (A, B, M_i, P_x, D_b) \end{cases} $$

$E_i$ and $E_j$ present implicit redundancy if and only if:

$$ \begin{cases} E_i = (A, B, M_i, P_x, D_\alpha) \\ E_j = (A, B, M_j, P_x, D_b) \\ M_i \text{ is an ancestor of } M_j, \end{cases} $$

where an ancestor can be a direct or an indirect parent.

Ontologies such as PSI-MI (Sivade Dumousseau et al., 2018) can be used to formalize the subsumption relations between IDMs. Note that with the notions provided in section 2.1, explicit and implicit redundancies might be observed among entries originating from different databases (inter-database redundancy, $D_\alpha \neq D_b$) but also from the same database (intra-database redundancy, $D_\alpha = D_b$). We will discuss later (Section 5.3) the meaning of intra-database redundancies, which can correspond either to multiple curation events, but also to variations of an IDM (for example, switching the experimental role ('bait' or 'prey') of the A and B proteins).

# 3 Methods

## 3.1 Source PPI datasets

PPI curation events integrated by APID were downloaded from the APID website on March 23, 2020, last update of APID in January, 2019) for two species (Homo sapiens and Saccharomyces cerevisiae) in the MITAB25 format (Kerrien et al., 2007). These files aggregate the curated events from five primary databases in a standard format.

In MITAB25 formated data, each line represents a curation event. Interacting proteins are identified by their Uniprot accession numbers. The organism is identified with its NCBI taxonomy identifier. Various information on the experimental evidence is also provided, notably the PMID of the source publication and the PSI-MI code of the IDM used

to detect the interaction. Some information such as the direction of the interaction (which protein was used as a 'bait' and which as a 'prey') is not available in this format, but it is usually recorded in primary databases or in more recent MITAB formats (MITAB27). If necessary, missing information might be retrieved using the primary database interaction identifier which is provided and offers full tractability.

## 3.2 RDF schema and triplestore

The global RDF schema used to integrate all information is presented in Figure 2. It relies on the following ontologies:

* Biological Pathway Exchange (BioPAX) is an ontology developed as a standard for representing molecular interactions, including protein-protein interactions (Demir et al., 2010). We followed the level 3 of the BioPAX specification.

* Proteomics Standards Initiative-Molecular Interactions (PSI-MI) is an ontology edited by the HuPO-PSI. It is dedicated to describe experimental IDMs (Sivade Dumousseau et al., 2018). We used version 1.2.

Raw PPI curation events from the MITAB file were first imported into a MySQL database. A Perl script was used to connect to this database, to exclude curation events that are not considered by APID (see below), and to convert it into a RDF dataset following the BioPAX v3 standard. The resulting interaction data were merged with the PSI-MI ontology, available as an OWL file, into a triplestore powered by the Apache Foundation's JENA suite (v3.14.0). The complete workflow is described in Supplementary Figure S1.

In its integration process, the APID meta-database does not consider curation events annotated with IDMs that do not correspond to a specific experimental method (Alonso-López et al., 2019). To be able to compare our results with APID, we also excluded from our analysis the very same curation events. These are the ones annotated with the IDMs 'molecular interaction', 'interaction detection method', 'biophysical', 'experimental interaction detection', 'inference', 'inferred by author', 'inferred by curator', 'in vitro', 'in vivo', 'unspecified method', or 'phenotype-based detection assay'.

## 3.3 SPARQL queries

Queries were run using SPARQL Protocol and RDF Query Language (SPARQL). The JENA suite was used to run the SPARQL queries. All queries used to detect redundancies are available in supplementary data
<figure><img src="image_66.png" /><figcaption>**Fig. 2.** Scheme representing two curation events reporting the interaction between the ubiquitin ligase MDM2 (UniprotKB: P04637) and the tumor protein 53 (UniprotKB: Q00987) in the BioPAX level 3 ontology (yellow nodes). These two curation events (highlighted in bold) were annotated by different databases (BioGRID and Intact). They refer to the same publication (PMID: 22819825), but different IDMs were used to annotate the interaction ('anti tag coimmunoprecipitation' and 'affinity chromatography technology'). The PSI-MI ontology (purple nodes) reveals that 'affinity chromatography technology' is an ancestor of 'anti tag coimmunoprecipitation', indicating an implicit redundancy between the two curation events.</figcaption></figure>

(Figures S2, S3, S4, S5, S6, S7). As an example, Figure 3 presents the SPARQL query used to detect implicit redundancies in curation events, if one term is an ancestor of the other in the PSI-MI ontology. For each implicit redundancy detected, we conserved only the curation event with the most precise IDM.

```sql
SELECT DISTINCT ?p1 ?p2 ?pmid ?dm_name1
WHERE {
    ?ppi1 rdf:type bp:MolecularInteraction;
    bp:participant ?p1, ?p2;
    bp:xref ?pmid;
    bp:evidence ?dm_name1.
    ?dm_name1 bp:evidenceCode ?m_vocab1.
    ?m_vocab1 bp:xref ?dm_code1.
FILTER (STR(?p1) < STR(?p2))
FILTER NOT EXISTS {
    ?ppi2 rdf:type bp:MolecularInteraction;
    bp:participant ?p1, ?p2;
    bp:xref ?pmid;
    bp:evidence ?dm_name2.
    ?dm_name2 bp:evidenceCode ?m_vocab2.
    ?m_vocab2 bp:xref ?dm_code2.
    ?dm_code2 rdfs:subClassOf+ ?dm_code1.
}
```

<caption>**Fig. 3.** SPARQL query to select curation events without explicit nor implicit redundancies.
(Note: prefixes are not shown)</caption>

## 3.4 Availability and implementation

The code is available at https://gitlab.com/nnet56/reproducible-interactome. The results are available at https://reproducible-interactome.genouest.org/ and on the Zenodo open data repository (doi:10.5281/zenodo.5595037). The non-redundant interactomes are also accessible on the NDEX platform to facilitate their analysis and manipulation with classical algorithms (doi:10.18119/N94302 (human), doi:10.18119/N97S4D (yeast)).

# 4 Results

## 4.1 Overview of analyzed curation events

We analysed the same curation events as the APID database to assess the efficiency of redundancy detection methods. A summary of these curation events is presented in Table 1. The downloaded MITAB files contain 700, 484 curation events for *Homo sapiens* and 305, 102 for *Saccharomyces cerevisiae* (hereinafter referred to as human and yeast, respectively). Together, BioGRID and IntAct represent approximately 85% of all curation events in both species. The contribution of HPRD and BioPlex, restricted to human data, accounts for 13.9% of human curation events. For both species, most PPIs appear in only one or two curation events. PPIs reported by a single curation event represent 49.3% and 60.7% of interacting pairs in human and yeast, respectively.

## 4.2 Interaction detection methods (IDMs)

The most frequent IDMs in all curation events are listed in Table 1. Among them, 'affinity chromatography technology', 'tandem affinity purification', 'anti tag coimmunoprecipitation' and 'two hybrid' cover more than 58% of human and 76% of yeast curation events. Interestingly, these IDMs include terms with parent-child relationships in the PSI-MI ontology. For example, 'affinity chromatography technology' is a direct ancestor of 'anti tag coimmunoprecipitation'. The presence of such chains is suggestive of possible implicit redundancies between curation events, as defined in sections 2.1 and 2.2.

## 4.3 Quantification of implicit redundancies

Thanks to the expressiveness of the SPARQL language, we identified both explicit and implicit redundancies among curation events (example query in Figure 3). For constituting a non-redundant dataset, we selected the most precise curation events and discard the redundant and less precise ones since they do not add information.

The occurrence of redundancy among curation events is significant (Table 2). We detected and discarded 73, 991 (11.1%) and 40, 266 (13.7%) implicitly redundant curation events for human and yeast, respectively. Taking into account both explicit and implicit redundancies resulted in removing 30.9% of curation events for human and 35.4% for yeast.
<table><caption>Table 1. Human and yeast curation events (CEs) analysed in this study. Excluded Interaction Detection Methods (IDMs) concern 5.00% (n = 35,000) of all curation events in human and 3.87% (n = 11,809) in yeast. Only IDMs annotated with a frequency higher than 2% are shown.</caption><thead><tr><th colspan="3">Contributing Databases</th><th colspan="3">Most frequent Interaction Detection Methods</th><th colspan="3">Curations events for (P<sub>a</sub>, P<sub>b</sub>)</th></tr><tr><th>Databases</th><th>CEs</th><th>(%)</th><th>Interaction Detection Methods</th><th>Counts</th><th>(%)</th><th>Occurrences</th><th>Counts</th><th>(%)</th></tr></thead><tbody><tr><td colspan="9"><strong>Human</strong></td></tr><tr><td>BioGRID</td><td>378,910</td><td>(54.1%)</td><td>Affinity chromatography technology</td><td>291,621</td><td>(41.63%)</td><td>One</td><td>161,031</td><td>(49.30%)</td></tr><tr><td>IntAct</td><td>215,577</td><td>(30.8%)</td><td>Two hybrid</td><td>71,969</td><td>(10.27%)</td><td>Two</td><td>91,742</td><td>(28.08%)</td></tr><tr><td>BIOPLEX!</td><td>55,151</td><td>(7.9%)</td><td>Anti tag coimmunoprecipitation</td><td>49,428</td><td>(7.06%)</td><td>[3-10]</td><td>69,015</td><td>(21.13%)</td></tr><tr><td>HPRD</td><td>42,327</td><td>(6.0%)</td><td>Pull down</td><td>42,423</td><td>(6.06%)</td><td>[10-50]</td><td>4,763</td><td>(1.46%)</td></tr><tr><td>DIP</td><td>8,519</td><td>(1.2%)</td><td>Biochemical</td><td>40,544</td><td>(5.79%)</td><td>≥ 50</td><td>113</td><td>(0.03%)</td></tr><tr><td colspan="9"></td></tr><tr><td colspan="9"><strong>Yeast</strong></td></tr><tr><td>BioGRID</td><td>133,998</td><td>(43.9%)</td><td>Affinity chromatography technology</td><td>88,681</td><td>(29.07%)</td><td>One</td><td>83,799</td><td>(60.73%)</td></tr><tr><td>IntAct</td><td>130,025</td><td>(42.6%)</td><td>Tandem affinity purification</td><td>84,842</td><td>(27.81%)</td><td>Two</td><td>28,496</td><td>(20.65%)</td></tr><tr><td>DIP</td><td>41,079</td><td>(13.5%)</td><td>Anti tag coimmunoprecipitation</td><td>35,363</td><td>(11.59%)</td><td>[3-10]</td><td>21,792</td><td>(15.79%)</td></tr><tr><td colspan="9"></td></tr><tr><td colspan="9"><strong>Apparent size of the reproducible interactome (PPIs supported by ≥ 2 EEs)</strong></td></tr><tr><td>Initial</td><td></td><td></td><td>Two hybrid</td><td>24,752</td><td>(8.11%)</td><td>[10-50]</td><td>3,799</td><td>(2.75%)</td></tr><tr><td>Without explicit redundancies</td><td></td><td></td><td>Pull down</td><td>13,960</td><td>(4.58%)</td><td>≥ 50</td><td>99</td><td>(0.07%)</td></tr><tr><td>Without explicit and implicit redundancies</td><td></td><td></td><td>Inferred by author</td><td>10,894</td><td>(3.57%)</td><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td><td>Protein complementation assay</td><td>6,825</td><td>(2.24%)</td><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td><td>Enzymatic study</td><td>6,817</td><td>(2.23%)</td><td></td><td></td><td></td></tr></tbody></table>

<table><caption>Table 2. Impact of the removal of both explicit and implicit redundancies on the number of curation events and on the apparent size of the reproducible interactome, for human and yeast. (EEs: Experimental Evidences)</caption><thead><tr><th></th><th>Human</th><th>(%)</th><th>Yeast</th><th>(%)</th></tr></thead><tbody><tr><td colspan="5"><strong>Curation events</strong></td></tr><tr><td>Initial curation events</td><td>665,484</td><td>(100%)</td><td>293,293</td><td>(100%)</td></tr><tr><td>Curation events without explicit redundancies</td><td>534,140</td><td>(80.3%)</td><td>229,630</td><td>(78.3%)</td></tr><tr><td>Curation events without explicit and implicit redundancies</td><td>460,149</td><td>(69.1%)</td><td>189,364</td><td>(64.6%)</td></tr><tr><td colspan="5"><strong>Apparent size of the reproducible interactome (PPIs supported by ≥ 2 EEs)</strong></td></tr><tr><td>Initial</td><td>159,192</td><td>(100%)</td><td>52,313</td><td>(100%)</td></tr><tr><td>Without explicit redundancies</td><td>111,009</td><td>(69.7%)</td><td>40,235</td><td>(76.9%)</td></tr><tr><td>Without explicit and implicit redundancies</td><td>70,554</td><td>(44.3%)</td><td>21,311</td><td>(40.7%)</td></tr></tbody></table>

Importantly, detection of redundancy between curation events has a strong impact on the apparent size of the reproducible interactome (i.e PPIs supported by at least two experimental evidences) (Table 2, Supplementary Figures S8 and S9). For human, the reproducible interactome drops from 159, 192 to 70, 554 PPIs (–55.7%: –30.3% due to explicit redundancies and –25.4% due to implicit ones). For yeast, the impact of redundancies is even worse, with a drop of the reproducible interactome from 52,313 PPIs to 21,311 after removal of both explicit and implicit redundancies (–59.3%: –23.1% due to explicit redundancies and –36.2% due to implicit ones). In other words, for human, discarding 11.1% of implicitly redundant curation events accounts for reducing by 25.4% the reproducible interactome. Similarly, for yeast, discarding 13.7% of implicitly redundant curation events accounts for reducing by 36.2% the reproducible interactome.

## 4.4 Implicit redundancies mostly result from the integration of the different primary databases

We then investigated whether implicit redundancy was already present in source databases (intra-database redundancy), or if it was a consequence of the integration of different source databases (inter-database redundancy). The vast majority originates from inter-database redundancies for both human (91.1%) and yeast (95.0%) (see Supplementary Tables S1 and S2). The couple of databases that generates the largest part of the implicit

redundancies is BioGRID and IntAct. This is consistent with the fact that BioGRID and IntAct are the two most contributing source databases. Intra-database redundancies will be further discussed in section 5.3.

## 4.5 Frequently redundant identification methods

We computed the frequency of the pairs of detection methods involved in implicit redundancies. For human, the most frequent implicitly redundant couples of IDMs and their parent-child relationships in the PSI-MI ontology are displayed in Figure 4.

The most frequent couple is 'affinity chromatography technology' and 'anti tag coimmunoprecipitation', which is responsible for 25, 333 redundancies. The term 'affinity chromatography technology' is also frequently observed with other descendants such as "pull down" (n = 9, 896), 'anti bait coimmunoprecipitation' (n = 6, 617), or "tandem affinity purification" (n = 5, 968). Two-hybrid techniques are also introducing redundancies, for example with 'two hybrid', and its descendants 'two hybrid array' (n = 16, 113), 'two hybrid prey polling approach' (n = 11, 238), 'validated two hybrid' (n = 11, 123), or 'two hybrid pooling approach' (n = 10, 713). A similar situation is observed in yeast (the complete list of implicit redundancies for both human and yeast is available as Supplementary Tables S3 and S4). Implicit redundancies are thus widespread all along the PSI-MI ontology, and not limited to binary IDMs. This highlights the need for a general approach to reconcile
<figure><img src="image_67.png" /><figcaption>**Fig. 4.** Couples of related interaction detection methods (IDMs) from the PSI-MI ontology frequently identified in implicit redundancies of human PPIs. The arrows connect the most specific to the most general term according to the PSI-MI ontology. Only implicit redundancies with at least 50 occurrences are shown. Nodes connected to a common IDM are represented with the same color.</figcaption></figure>

curation events during the integration of multiple primary databases.

The fact that implicit redundancies are observed between very different terms of the PSI-MI ontology suggests that different primary databases have different policies for annotating IDMs, as previously noted for IntAct and BioGRID (Alonso-López et al., 2019). We therefore further analysed the IDMs used by each primary database.

We observed that IntAct and DIP use a wide range of IDMs for both human and yeast PPIs (165 for IntAct and 89 for DIP) while BioGRID, HPRD and BioPlex use much fewer (12, 3 and 1 IDMs, respectively) and more general IDMs. Hence, the strong discrepancies in database annotation policies are the source of inter-database implicit redundancies.

Overall, we observed that implicit redundancy (i) occurs between a wide range of the PSI-MI ontology terms, regardless of the species, (ii) mostly results from the integration of different primary databases with different annotation policies, and (iii) happens for all database combinations.

## 5 Discussion

The construction of a reliable interactome demands to combine interaction data produced by several independent experimental evidences and IDMs in order to reduce false positives. Since experimental evidences are curated and stored in several primary databases, a unification of these databases is required. The Human Proteome Organization Proteomics Standards Initiative (HUPO-PSI) developed the PSICQUIC specification and web services that facilitate data retrieval from multiple databases and assist their integration but do not elaborate on redundancy detection (del Toro et al., 2013). In several (meta-) databases, PPIs are annotated with a confidence score, which is calculated using the number of independent experimental evidences and the nature of IDMs (Villaveces et al., 2015). To be relevant, these algorithmic require reliable, non-redundant, datasets of experimental evidences. Therefore, several primary databases have decided to coordinate their curation efforts in the frame of the IMEx consortium in order to provide a single non-redundant set of homogeneously annotated protein interaction data (Orchard et al., 2012; Porras et al., 2020).

Here, we propose a formalisation of both explicit and implicit redundancy between experimental evidence entries in order to integrate PPIs from any database that uses the PSI-MI ontology. Knowledge about

IDMs is extracted from the PSI-MI ontology, while the method to identify redundancies is based on Semantic Web technologies.

### 5.1 The Semantic Web is adapted for identifying implicit redundancies

Alonso-López et al. (2019) pointed two problems related to redundancy identification: (i) there may be a parent-child relationship between IDM terms, and (ii) the path from a child term to its ancestors may not be unique due to multiple inheritance. We propose the notion of **implicit redundancy** to address the logical implications of two database entries describing the interaction of the same protein pair with IDMs that have a descendant-ancestor relationship. The Semantic Web is designed to perform integrated reasoning on data annotations and ontologies. In particular, it makes handling simple and multiple hierarchies straightforward. In the raw data of APID that aggregates BioGRID, IntAct, HPRD, BioPlex and DIP, we were able to identify both explicit and implicit redundancies. Our work reveals that implicit redundancies are a widespread phenomenon resulting from the different curation choices of the various databases and that it is of similar importance than explicit redundancies. Therefore, we demonstrated the relevance of both the notion of implicit redundancy and of the choice of the Semantic Web as a technical framework for addressing the redundancy identification problem. Moreover, new explicit and implicit redundancies will continue to occur over the natural updates of the various databases.

The PSI-MI ontology that describes the IDMs is evolving. For example, during the time of our project, we noticed that the term 'three hybrid', which was initially a child of the term 'two hybrid', is now a child of 'transcriptional complementation assay'. This modification is highly relevant since 'two hybrid' is a binary identification method, whereas 'three hybrid' is not, and having a non-binary identification method as a direct child of a binary one was not consistent. Therefore, just like the databases are regularly updated, the ontologies are also corrected and enriched, which also has an incidence on redundancies. By allowing to automate redundancy detection as the integration of databases scales up, the Semantic Web facilitates the reliable interpretation of the results in the perspective of the construction of a reproducible interactome.

### 5.2 Widespread inter-databases implicit redundancies

Implicit redundancies primarily arise from the integration of different databases (91.1% and 95.0% of inter-database redundancies for human
and yeast, respectively). In our study, we clearly highlight that this is due to the granularity of IDMs used in the primary databases. Indeed, while some databases like IntAct refer to numerous detailed terms from the PSI- MI ontology (165 and 89 terms used to annotate human and yeast PPIs, respectively), other databases like BioGRID merely use general and high level terms (only 12 terms used for both human and yeast).

Therefore, if the integration of different PPI databases is necessary to better cover the interactome, a particular attention has to be paid to detect the widespread inter-database implicit redundancies. A simple method could be to define priorities between databases depending on whether they use precise or general terms to annotate PPIs. In case of multiple curations events referring to the same proteins and the same PMID, the ones from the database with the highest priority would be selected. However, this would be an approximate approach whereas we propose an exact solution, robust to possible changes of annotation policy by primary databases.

Primary databases of the IMEx consortium coordinate and share their curation efforts to produce a non-redundant dataset of PPI experimental evidences (Orchard et al., 2012). IMEx members use common curation rules to harmonize their annotation process. The unicity of the curation events is ensured by allowing PPIs from a given PMID to be annotated only once, and all data are centralized in IntAct. Both this work from the IMEx consortium and ours emphasize the need for a general approach to assemble non-redundant PPI datasets.

5.3 Intra-database redundancies

Our analysis also identified a significant number of apparently redundant curation events within primary databases (Supplementary Figures S8 and S9). Such intra-database redundancy may originate from multiple independent annotations of identical experimental evidences within primary databases, as noted by Alonso-López et al. (2019). Yet, further inspection of such curation events indicates that intra-database redundancy primarily occurs when independent experiments from the same publication have been annotated in a given database with identical or related IDMs, leading to apparent explicit or implicit intra-database redundancies. For instance, we observed that the vast majority of the explicit intra-database redundancies originating from BioGRID are due to PPIs probed with both partners as baits and preys (6229 out of 8696 explicit redundancies involving exactly two curation events for yeast and 12283 out of 15385 for human). Intra-database redundancy can also occur when a PPI has been identified with a high-throughput experiment and then validated using the same or a related method performed at low-throughput. Hence, this currently leads to the unification of curation events that actually report distinct experimental evidences. To correct this, our method could be extended by taking into account additional information, such as the experimental role of each protein.

5.4 Towards a reproducible interactome

The size of the reproducible interactome is drastically impacted by removing redundancies for both human (−55.7%) and yeast (−59.3%), and we show that this is largely due to implicit redundancies. Indeed, we observe that filtering the curation events involved in implicit redundancy (11 to 14 %) leads to a drastic (25 to 36 %) reduction of the apparently reproducible interactome. This implies that a large number of PPIs currently considered as reproducible actually relies on integration artefacts. Thus, more experimental data are still needed to further improve the size and confidence level of the reproducible interactome. Information on PPIs that have not yet been reproduced can help to prioritize such experiments. Knowledge-based methods as presented in this article will be necessary to support the integration of the continuously increasing experimental evidences and publications.

Acknowledgements

The GenOuest platform provided computational support and Web hosting.

Funding

This work has been supported by Univ Rennes with a Defi Emergent 2019 grant to EB and GR.

References

Alonso-López, D. et al. (2016). APID interactomes: providing proteome-based interactomes with controlled quality for multiple species and derived networks. *Nucleic Acids Res.* **44**(W1), W529–535.

Alonso-López, D. et al. (2019). APID database: redefining protein–protein interaction experimental evidences and binary interactomes. *Database*, **2019**, baz005.

Bonetta, L. (2010). Interactome under construction. *Nature*, **468**(7325), 851–852.

Cafarelli, T. et al. (2017). Mapping, modeling, and characterization of protein–protein interactions on a proteomic scale. *Current Opinion in Structural Biology*, **44**, 201–210.

Calderone, A. et al. (2013). mentha: a resource for browsing integrated protein- interaction networks. *Nat Methods*, **10**(8), 690–691.

Calderone, A. et al. (2020). Using the MINT Database to Search Protein Interactions. *Curr Protoc Bioinformatics*, **69**(1), e93.

Das, J. and Yu, H. (2012). HINT: High-quality protein interactomes and their applications in understanding human disease. *BMC Syst Biol.*, **6**, 92.

del Toro, N. et al. (2013). A new reference implementation of the PSICQUIC web service. *Nucleic Acids Res.*, **41**(Web Server issue), W601–606.

Demir, E. et al. (2010). The BioPAX community standard for pathway data sharing. *Nature biotechnology*, **28**(9), 935–942.

Hutlin, E. L. et al. (2021). Dual proteome-scale networks reveal cell-specific remodeling of the human interactome. *Cell*, **184**(11), 3022–3040.

Kerrien, S. et al. (2007). Broadening the horizon–level 2.5 of the HuPO-PSI format for molecular interactions. *BMC Biol*, **5**, 44.

Kerrien, S. et al. (2012). The IntAct molecular interaction database in 2012. *Nucleic Acids Res*, **40**(Database issue), D841–846.

Keshava Prasad, T. S. et al. (2009). Human Protein Reference Database–2009 update. *Nucleic Acids Res*, **37**(Database issue), D767–772.

Klapa, M. I. et al. (2013). Reconstruction of the experimentally supported human protein interactome: what can we learn? *BMC systems biology*, **7**, 96.

Kotlyar, M. et al. (2019). IID 2018 update: context-specific physical protein-protein interactions in human, model organisms and domesticated species. *Nucleic Acids Res*, **47**(D1), D581–D589.

Luck, K. et al. (2020). A reference map of the human binary protein interactome. *Nature*, **580**(7803), 402–408.

Orchard, S. et al. (2012). Protein interaction data curation: the International Molecular Exchange (IMEx) consortium. *Nat Methods*, **9**(4), 345–350.

Oughtrid, R. et al. (2019). The BioGRID interaction database: 2019 update. *Nucleic Acids Res*, **47**(D1), D529–D541.

Porras, P. et al. (2020). Towards a unified open access dataset of molecular interactions. *Nature communications*, **11**(1), 6144.

Salwinski, L. et al. (2004). The Database of Interacting Proteins: 2004 update. *Nucleic Acids Res*, **32**(Database issue), D449–451.

Sivade Dumousseau, M. et al. (2018). Encompassing new use cases - level 3.0 of the HuPO-PSI format for molecular interactions. *BMC Bioinformatics*, **19**(1), 134.

Szklarczyk, D. et al. (2019). STRING v11: protein-protein association networks with increased coverage, supporting functional discovery in genome-wide experimental datasets. *Nucleic Acids Res*, **47**(D1), D607–D613.

Turinsky, A. L. et al. (2010). Literature curation of protein interactions: measuring agreement across major public databases. *Database*, **2010**, baq026.

Villaveces, J. M. et al. (2015). Merging and scoring molecular interactions utilising existing community standards: tools, use-cases and a case study. *Database*, **2015**, bau131.
# Detection and correction of non-conformities and redundancies in complexes of molecules in BioPAX

Camille JUIGNÉ<fnref n="1" />,², Olivier DAMERON<fnref n="1" />, François MOREEWS<fnref n="1" />,², Florence GONDRET² and
Emmanuelle BECKER<fnref n="1" />

¹ Univ Rennes, Inria, CNRS, IRISA - UMR 6074, F-35000 Rennes, France

² PEGASE, INRAE, Institut Agro, 35590, Saint Gilles, France

Corresponding author: camille.juigne@irisa.fr

**Abstract** Complexes play major roles in the interactions between biochemical entities and the regulation of biological pathways. The Biological Pathway Exchange format (BioPAX) facilitates the integration of data sources describing such interactions. In BioPAX, a black-box complex is a complex with no component, as its composition is unknown. The BioPAX specification explicitly prevents complexes to have any component that is another complex unless this component is a black-box complex. We observed that the Reactome pathway database contains such invalid recursive complexes of complexes. We propose a method to identify and fix them. For the Homo sapiens version of Reactome, 5,734 of the 14,840 complexes (39%) are invalid. They participate to 6,994 of the 21,813 interactions (32%). All the complexes identified as invalid could be fixed. In addition to improving BioPAX compliance, the benefits of fixing invalid recursive complexes are two-folds. First, it led to a decrease in false negative when identifying the interactions in which a physical entity can participate as a component of a complex. With the original Reactome, a set of proteins of interest participated in 554 interactions either directly or as direct parts of complexes. Fixing invalid recursive complexes allowed to identify 429 additional interactions (+77%). Second, it decreased artificial redundancies in Reactome by identifying sets of structurally equivalent complexes that share the same components with the same stoichiometric coefficients and have the same cellular location. The original Reactome contained 241 redundant complexes. Fixing invalid recursive complexes allowed to identify 92 additional redundant complexes (+38%).

**Keywords** BioPAX, Curation, Molecular complex, SPARQL, Reactome

## 1 Introduction

*Complexes and interactions in biology.* Understanding how biological systems adapt to their environment needs to better capture, describe and model the interactions between their biochemical entities. A rigorous approach is required to support further analyses because these systems typically involve an intricate network of interactions between numerous participants. Among them, complexes are a major class of entities that results from the chemical assembly of several molecules (nucleic acids, proteins and other molecules) that bind each other at the same time and place, and form single multimolecular machines. Biologically, they play an important role in transcription, RNA splicing and polyadenylation machinery, protein export and transport etc [1,2]. From the data analysis perspective, they cause some indirection between molecules and interactions, i.e. this introduces an additional node (the complex) between two entities that are no longer directly connected by a link. Therefore, molecules can either participate directly to an interaction or also be a component of a complex that participates to an interaction. The latter should also be considered.

*Complexes in BioPAX.* The Biological Pathway Exchange format<fnref n="1" /> (BioPAX) is a well established formalism to represent biological pathways at the molecular and cellular levels, including interactions [3]. In the BioPAX ontology, the four top level classes are **Pathway**, **Interaction**, **Physical Entity** and **Gene**. Interactions represent biological relationships between two or more entities. They encompass molecular interactions, controls and conversions. Physical entities encompass small molecules,

<footnote n="1">1. http://www.biopax.org/release/biopax-level3-documentation.pdf</footnote>
proteins, DNA, RNA and complexes. Complexes are defined in BioPAX as “*physical entities whose structure is comprised of other physical entities bound to each other non-covalently, at least one of which is a macromolecule (e.g. protein, DNA, or RNA)*”. If BioPAX requires that at least one of the components of a complex is a macromolecules, the specification also addresses the situation where another component would be a complex. It explicitely mentions: “*complexes should not be defined recursively [...] i.e. a complex should not be a component of another complex. [...] Exceptions are black-box complexes (i.e. complexes in which the component property is empty), which may be used as components of other complexes because their constituent parts are unknown*”.

*The Reactome use-case.* BioPAX is based on Semantic Web technologies, with RDF facilitating integration, SPARQL facilitating querying and OWL facilitating knowledge-based reasoning. All the major reference pathway databases are available in BioPAX. Among them, Reactome<fnref n="2" /> is a free, open-source, curated and peer-reviewed pathway database [4]. It is widely used in genome analysis, modeling, systems biology, clinical research and education. We noticed the presence of invalid recursive complexes, i.e. complexes composed of other complexes that are not black-box complexes. Interestingly, these invalid complexes were not detected by the BioPAX validator<fnref n="3" /> [5]. Their presence in Reactome precludes further robust analyses.

## 2 Objectives

Our hypothesis is that recursive complexes can cause false negatives when identifying the interactions in which a physical entity can participate. For example, if A, B and C are physical entities, A can directly participate in several interactions, but also indirectly when associated to B as a complex, or to B and C as another complex. If (A,B) and (A,B,C) are valid complexes, the two situations can be correctly processed by identifying the interactions matching with the criterion “having a participant that is a complex composed of A”. However, if (A,B,C) complex is composed of (A,B) complex and of C, all the interactions in which (A,B,C) participates would fail to meet the aforementioned criterion because their participants are not “a complex composed of A” but “a complex composed of a complex composed of A”.

We propose a method for identifying and fixing these invalid recursive complexes. Using this method, we determine whether fixing invalid recursive complexes, thereby repairing the topology of the graph, is beneficial by allowing to identify new interactions in which a molecule participates.

In addition, fixing invalid recursive complexes could also result in the identification of redundant complexes. For example, ((A,B),C) or (A,(B,C)) or ((A,C),B) could be distinct invalid complexes with their own identifiers, but they would be all fixed as complexes having the same components A, B and C using our method. We also study whether some of the invalid complexes that were fixed at the previous step become equivalent to other fixed invalid complexes or to original valid complexes.

## 3 Materials and methods

We developed semantically-rich SPARQL queries for identifying and fixing invalid recursive complexes, and to detect the resulting redundancies. We applied this method on the Reactome pathways database as a use-case study. For the sake of reproducibility, we provide a jupyter notebook detailing the analysis<fnref n="4" />.

### 3.1 Identify invalid complexes

We first analyzed all the configurations of BioPAX complexes according to the nature of their components. Figure 1 summarizes the BioPAX validity of the different categories. We wrote a SPARQL query for each category to compute their relative size out of the total number of complexes. The query in figure 2 identifies the invalid recursive complexes.

<footnote n="2">2. https://reactome.org/</footnote>

<footnote n="3">3. https://biopax.baderlab.org/</footnote>

<footnote n="4">4. https://github.com/cjuigne/non_conformities_detection_biolpax</footnote>
<figure><img src="image_68.png" /><figcaption>**Fig. 1.** Validity of the categories of BioPAX complexes. Complexes are represented by circles and the other physical entities by boxes with rounded corners. Composition is represented by a diamond head arrow with the diamond on the side of the whole. A valid complex can have components that are themselves complexes only if they are all black-box complexes, i.e. they do not have any component. Note that an invalid complex can itself be a component of another complex (which therefore becomes invalid as well).</figcaption></figure>

```sparql
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX bp3: <http://www.biopax.org/release/biopax-level3.owl#>
```

```sql
SELECT DISTINCT ?invalidComplex
WHERE {
    ?invalidComplex rdf:type bp3:Complex .
    ?invalidComplex bp3:component ?complexComponent .
    ?complexComponent rdf:type bp3:Complex .
    ?complexComponent bp3:component ?componentOfComplexComponent .
}
```

<caption>**Fig. 2.** SPARQL query for identifying invalid recursive complexes in BioPAX, i.e. complexes composed of at least another complex that has components.</caption>

## 3.2 Fix the invalid complexes

Fixing an invalid recursive complex consists in (1) collapsing as direct components all its direct or indirect atomic components (i.e. either not complexes or black-box complexes; they correspond to the leaves in the tree of components), (2) deleting all the other components, (3) setting the correct values for stoichiometric coefficients, and (4) preserving all its other attributes.

For computing the values of the stoichiometric coefficients in step (3), we need to trace all the stoichiometric coefficients from each leaf up to the root complex, and to take into account the fact that a given physical entity can be a component of several parts of the recursive complex. Tracing stoichiometry is illustrated by figure 3 where complex Y was composed of $\gamma$ Z, and X was itself composed of $\alpha$ Y. This resulted in $\alpha \times \gamma$ Z in X (via Y). The second situation is illustrated by the
fact that, in addition to being a component of Y, Z was also a direct component of X with β as stoichiometric coefficient value. Overall, this resulted in α × γ + β occurrences of Z in X.

We note $S_y(z)$ the global stoichiometric coefficient value of $z$ at $y$, i.e. the number of occurrences of $z$ in $y$, and $C(y)$ the set of the direct components of $y$. Formula 1 recursively computes the stoichiometric coefficient value of any physical entity $z$.

$$ \left\{ \begin{array}{ll} S_z(z) = 1 & \text{if } (y \neq z) \land (C(y) = \emptyset) \\ S_y(z) = 0 & \text{otherwise} \\ S_y(z) = \sum_{p \in C(y)} S_y(p) \times S_p(z) & \end{array} \right. \qquad (1) $$

<figure><img src="image_69.png" /><figcaption>Fig. 3. Fixing an invalid recursive complex consists in collapsing as direct components all its direct and indirect components that are leaves in the composition tree of the complex and computing the correct stoichiometric coefficient values with equation 1.</figcaption></figure>

## 3.3 Identify redundant complexes

We considered as redundant the complexes that have exactly the same components with the same stoichiometric values and the same cellular location. Figure 4 illustrates how invalid recursive complexes can be the cause of redundancy due to the order by which the components are nested in the complexes. Fixing the invalid complexes made possible the detection of redundancy. For that, we developed a SPARQL query that identifies the pairs of complexes that have the same components and properties but different identifiers.

<figure><img src="image_70.png" /><figcaption>Fig. 4. Example of invalid recursive complexes leading to redundancy. Fixing them made possible the detection of redundancy.</figcaption></figure>
## 3.4 Application to Reactome

We used the Reactome pathway database (Homo sapiens version 79 (2022-02-03)) as a use-case study. First, we identified the invalid recursive complexes, the molecules they were composed of, and the interactions they participated in. Second, we fixed the invalid recursive complexes and generated their valid representation. Third we generated a corrected version of Reactome with no invalid recursive complexes: we deleted the composition relations and the stoichiometric coefficients of the invalid complexes, and then imported their valid representation.

## 3.5 Evaluation for determining the reactions a molecule participates to

Complexes induce more than one level of indirection between biological entities and the reactions to which they participate (as products, substrates, activators or inhibitors).

1. From a list of molecules (ChEBI ID or UniProt ID) we located the corresponding entities

- for each molecule we looked for the corresponding entity reference, which in BioPAX means a node where all the non-changing aspects of the entity are stored.

- then, we identify all the physical entities associated to these entity references. Having several physical entities associated to a single entity reference allows to associated them with different contexts such as cellular location.

2. From these physical entities, we identified the interactions in which they participate, including as a component of a valid or invalid complex:

- as direct participant of an interaction

- before fixing invalid complexes, the interactions that had a complex in which the molecule was a direct component

- after fixing invalid complexes, the interactions that have a complex in which the molecule is a *direct* component. Note that this corresponds to the interactions that have a participant that is an invalid complex in which the molecule is an *indirect* component.

# 4 Results

## 4.1 Invalid complexes identified in Reactome

The Human subset of Reactome v79 is composed of a total of 14,840 complexes. Among them, we identified 858 black-box complexes. Among the remaining 13,982 complexes (that had at least one known component), 8,248 complexes had no component that was itself a complex with known components. Together with the black-box complexes, they represented the 9,106 valid complexes. On the opposite, 5,734 complexes had at least one component that was a complex with components. Altogether, invalid recursive complexes represented 39% of the 14,840 complexes in Reactome. None of them have been detected by the BioPAX-validator tool.

Invalid recursive complex participated to 6,994 of the 21,813 interactions (32%).

## 4.2 Fixed invalid complexes

All the 5,734 complexes were fixed.

In the Reactome dataset, the average number of direct components in a complex was 2 ($\sigma$ = 2.6), with R-HSA-5626171 and R-HSA-72069 having each a maximum of 65 components. After fixing the invalid complexes, the average number of direct components was 4 ($\sigma$ = 8.7), with R-HSA-156656 having the maximum of 174 components. Figure 5 illustrates the number of direct components counted before and after fixing invalid complexes. Indeed, fixing the invalid complexes allowed to repair the topology of the graph.

## 4.3 Identified redundant complexes

Among the 14,840 complexes from the original Reactome, the query identified 133 pairs of redundant complexes involving 241 distinct complexes. They constituted 117 maximal cliques of equivalent complexes (size ranging from 2 to 4), corresponding to 124 complexes in excess.
<figure><img src="image_71.png" /><figcaption>**Fig. 5.** Number of direct components of the complexes in the Human dataset of Reactome v79 before (top) and after (bottom) fixing invalid recursive complexes.</figcaption></figure>

After fixing the invalid complexes, the query identified 210 pairs of redundant complexes involving 333 distinct complexes. They constituted 157 maximal cliques of equivalent complexes (size ranging from 2 to 6), corresponding to 176 complexes in excess. Figure 6 illustrates how fixing structurally different complexes revealed their redundancy.

The files reactome-v79-complexes-orig-redundant.ttl and reactome-v79-complexes-valid-redundant.ttl contain symmetric dyliss:hasSameCompositionAs relations between pairs of redundant complexes.

<figure><img src="image_72.png" /><figcaption>**Fig. 6.** Original compositions of complexes R-HSA-9670369 (top left) and R-HSA-9670372 (top right) from Reactome. Both are invalid recursive complexes. The fixed versions (bottom left and right, respectively) have more direct components than the original. Both fixed versions have the same components with the same stoichiometric coefficients, which reveals their redundancy. The structure difference between the original versions is highlighted in red: R-HSA-9670372 is composed of an intermediate dimer of the p-STATs protein whereas R-HSA-9670369 is directly composed of p-STATs with a stoichiometric coefficient of 2.</figcaption></figure>

## 4.4 Evaluation of the benefits of the complete procedure

Complexes play an important role in Reactome, and the components of invalid recursive complexes participate in many interactions. When retrieving the interactions molecules participate to, we quantified the false negatives caused by invalid complexes and that become true positives after fixing the
invalid complexes, both in general and in an experimental use-case. As shown in table 1, complexes are the most common interaction participants in Reactome. The 5,734 invalid complexes were composed of 7,119 molecules (proteins, small molecules and physical entities). These molecules were direct participants of 7,832 interactions. They were direct participants or *direct* component of complexes that were direct participants of 12,148 interactions. The 55% increase demonstrates the relevance of considering complexes. They were direct participants or *direct* or *indirect* (which amounts to fixing the invalid recursive complexes) component of complexes that were direct participants of 13,211 interactions. The additional 9% increase demonstrates the benefit of fixing invalid complexes. Therefore, these 7,119 molecules participate to 13,211 interactions, 1063 (9%) of which are false negative caused by invalid complexes.

<table><caption>**Tab. 1.** BioPAX Types of interaction participants in Reactome Homo sapiens v79.</caption><thead><tr><th>Type of interaction participants</th><th>Total number of participants</th></tr></thead><tbody><tr><td><strong>Complex</strong></td><td><strong>10560</strong></td></tr><tr><td>Protein</td><td>7558</td></tr><tr><td>Biochemical reaction</td><td>7128</td></tr><tr><td>Small molecule</td><td>2980</td></tr><tr><td>Physical entity</td><td>1157</td></tr><tr><td>DNA</td><td>661</td></tr><tr><td>RNA</td><td>163</td></tr><tr><td>Degradation</td><td>7</td></tr><tr><td>Template reaction</td><td>6</td></tr></tbody></table>

To evaluate the benefit of the procedure, we considered eight modules of 7 to 67 unique transcripts of genes (Table 2). These modules were initially identified from an experimental microarray thanks to weighted gene co-expression network analysis [6] describing the correlation patterns among genes across microarray samples, and related to a phenotype of interest in animal production. From these experimental modules, we wanted to retrieve all the interactions associated to one of these genes (via their transcripts or encoded proteins). We compared the number of interactions found when identifying proteins from one of the modules as a direct participant, as the direct component of a complex that was a direct participant (no correction) and as the direct component of a complex after fixing and validation (i.e. by considering invalid complexes). Table 2 shows that more interactions were extracted after fixing the invalid complexes than when considering only the atomic entities, and even more when considering invalid complexes, i.e., using complex SPARQL paths through the *component* property of complexes to obtain their final components.

<table><caption>**Tab. 2.** Number of interactions found for each *module* composed of *ID UniProt* proteins by queries. Query1: Interactions these proteins are direct components. Query2: Interactions that have a participant of which the molecule is a direct component. Query3: Interactions that have a participant that is an invalid complex of which the molecule is an indirect component.</caption><thead><tr><th>Module</th><th>ID</th><th>UniProt</th><th>Corresponding ProteinReference</th><th>Corresponding Protein</th><th>Corresponding Interactions (Q1)</th><th>Corresponding Interactions (Q2)</th><th>Corresponding Interactions (Q3)</th><th>Gain</th></tr></thead><tbody><tr><td>module (1)</td><td>96</td><td>67</td><td></td><td>129</td><td>58</td><td>164</td><td>238</td><td>180</td></tr><tr><td>module (2)</td><td>44</td><td>27</td><td></td><td>32</td><td>20</td><td>73</td><td>157</td><td>137</td></tr><tr><td>module (3)</td><td>22</td><td>12</td><td></td><td>59</td><td>6</td><td>8</td><td>15</td><td>9</td></tr><tr><td>module (4)</td><td>14</td><td>10</td><td></td><td>14</td><td>5</td><td>13</td><td>89</td><td>84</td></tr><tr><td>module (5)</td><td>11</td><td>9</td><td></td><td>39</td><td>6</td><td>12</td><td>24</td><td>18</td></tr><tr><td>module (6)</td><td>10</td><td>7</td><td></td><td>18</td><td>3</td><td>70</td><td>120</td><td>117</td></tr><tr><td>module (7)</td><td>9</td><td>9</td><td></td><td>22</td><td>7</td><td>77</td><td>79</td><td>72</td></tr><tr><td>module (8)</td><td>7</td><td>49</td><td></td><td>200</td><td>71</td><td>137</td><td>261</td><td>190</td></tr></tbody></table>

# 5 Discussion

Invalid recursive complexes affect a large portion of Reactome. They constitute 39% of the complexes and participate in 32% of the interactions. They are not detected by the BioPAX validator. Therefore, it is crucial to address this problem.
The procedure developed herein to fix invalid complexes and identify redundant complexes helped to correct non-compliance with BioPAX specifications. By doing so, it also allowed to simplify any further analyses of biological networks, as recursive complexes that should not have existed can now be processed without having to modify standard queries or scripts based on BioPAX libraries such as Paxtools [7] or PyBioPAX [8].

As a first consequence, fixing invalid complexes allowed to decrease the number of false negative when retrieving the interactions to which a molecule of interest participates.

As a second consequence, fixing invalid complexes revealed redundancies that would have been difficult to identify. As Reactome contains mappings to ComplexPortal [9], we could suppose that redundant complexes are associated to the same ComplexPortal identifier. Future work will assess (1) whether all the complexes that are associated to the same ComplexPortal identifier were recognized as redundant with our method, and (2) whether all our predicted redundancies are confirmed by ComplexPortal (and if they are not, we could propose new mappings). However, ComplexPortal seems to incompletely support stoichiometry, which may be a severe limitation.

After addressing the question of the mappings to ComplexPortal, we will study whether our approach is applicable to other resources. The first step will confirm that our queries are compatible with any BioPAX-compliant database. The second step will assess the importance of invalid recursive complexes in the other species of Reactome and in the other major BioPAX pathway databases (Kegg, MetaCYC, wikipathways). Eventually, a longer-term perspective will be to study whether our redundancy queries are relevant for generating mappings when combining several BioPAX databases.

## References

[1] Javad Zahiri, Abbasali Emamjomeh, Samaneh Bagheri, Asma Ivazeh, Ghasem Mahdevar, Hessam Sepasi Tehrani, Mehdi Mirzaie, Barat Ali Fakheri, and Morteza Mohammad-Noori. Protein complex prediction: A survey. *Genomics*, 112(1):174-183, 2019.

[2] Victor Spirin and Leonid A Mirny. Protein complexes and functional modules in molecular networks. *Proceedings of the National Academy of Sciences of the United States of America*, 100(21):12123-12128, 2003.

[3] Emek Demir, Michael P Cary, Suzanne Paley, Ken Fukuda, and et al. The BioPAX community standard for pathway data sharing. *Nature biotechnology*, 28(9):935-942, 2010.

[4] Marc Gillespie, Bijay Jassal, Ralf Stephan, Marija Milacic, Karen Rothfels, Andrea Senff-Ribeiro, Johannes Griss, Cristoffer Sevilla, Lisa Matthews, Chuqiao Gong, Chuan Deng, Thawfeek Varusai, Eliot Ragueneau, Yusra Haider, Bruce May, Veronica Shamovsky, Joel Weiser, Timothy Brunson, Nasim Sanati, Liam Beckman, Xiang Shao, Antonio Fabregat, Konstantinos Sidiropoulos, Julieth Murillo, Guilherme Viteri, Justin Cook, Solomon Shorser, Gary Bader, Emek Demir, Chris Sander, Robin Haw, Guanming Wu, Lincoln Stein, Henning Hermjakob, and Peter D'Eustachio. The reactome pathway knowledgebase 2022. *Nucleic acids research*, 2021. In press.

[5] Igor Rodchenkov, Emek Demir, Chris Sander, and Gary D. Bader. The BioPAX Validator. *Bioinformatics* (*Oxford, England*), 29(20):2659-2660, October 2013.

[6] Peter Langfelder and Steve Horvath. WGCNA: an R package for weighted correlation network analysis. *BMC Bioinformatics*, 9(1):559, December 2008.

[7] Emek Demir, Ozgün Babur, Igor Rodchenkov, Bülent Arman Aksoy, Ken I Fukuda, Benjamin Gross, Onur Selçuk Sümer, Gary D Bader, and Chris Sander. Using biological pathway data with paxtools. *PLoS computational biology*, 9(9):e1003194, 2013.

[8] Benjamin M. Gyori and Charles Tapley Hoyt. PyBioPAX: biological pathway exchange in python. *Journal of Open Source Software*, 7(71):4136, 2022.

[9] Birgit H M Meldal, Livia Perfetto, Colin Combe, Tiago Lubiana, João Vitor Ferreira Cavalcante, Hema Bye-A-Jee, Andra Waagmeester, Noemi Del-Toro, Anjali Shrivastava, Elisabeth Barrera, Edith Wong, Bernhard Mlecnik, Gabriela Bindea, Kalpana Panneerselvam, Egon Willighagen, Juri Rappsilber, Pablo Porras, Henning Hermjakob, and Sandra Orchard. Complex portal 2022: new curation frontiers. 2021. In press.
Bibliography

Aksman, L. M., M. A. Scelsi, A. F. Marquand, D. C. Alexander, S. Ourselin, A. Alt-
mann, and for ADNI (Sept. 2019). "Modeling longitudinal imaging biomarkers
with parametric Bayesian multi-task learning". In: *Human Brain Mapping* 40.13,
pp. 3982-4000. DOI: 10.1002/hbm.24682.

Alanis-Lobato, G., M. A. Andrade-Navarro, and M. H. Schaefer (Jan. 2017). "HIP-PIE v2.0: enhancing meaningfulness and reliability of protein-protein interaction networks". In: *Nucleic Acids Res.* 45.D1, pp. D408-D414.

Alonso-López, D., M. A. Gutiérrez, K. P. Lopes, C. Prieto, R. Santamaría, and J. De Las Rivas (2016). "APID interactomes: providing proteome-based interactomes with controlled quality for multiple species and derived networks". In: *Nucleic Acids Res* 44.W1, W529-535.

Alonso-López, D., F. J. Campos-Laborie, M. A. Gutiérrez, L. Lambourne, M. A. Calder-
wood, M. Vidal, and J. De Las Rivas (Jan. 2019). "APID database: redefining pro-
tein-protein interaction experimental evidences and binary interactomes". In:
*Database* 2019. DOI: 10.1093/database/baz005. eprint: https://academic.oup.com/database/article-pdf/doi/10.1093/database/baz005/27644182/baz005.pdf.

Anders, S. and W. Huber (Oct. 2010). "Differential expression analysis for sequence count data". In: *Genome Biol.* 11.10, R106.

Antelmi, L., N. Ayache, P. Robert, and M. Lorenzi (May 2019). "Sparse Multi-Channel Variational Autoencoder for the Joint Analysis of Heterogeneous Data". In: *International Conference on Machine Learning*. PMLR, pp. 302–311.

Aranda, B., H. Blankenburg, S. Kerrien, F. S. L. Brinkman, A. Ceol, E. Chautard, J. M. Dana, J. De Las Rivas, M. Dumousseau, E. Galeota, A. Gaulton, J. Goll, R. E. W. Hancock, R. Isserlin, R. C. Jimenez, J. Kerssemakers, J. Khadake, D. J. Lynn, M. Michaut, G. O’Kelly, K. Ono, S. Orchard, C. Prieto, S. Razick, O. Rigina, L. Salwinski, M. Simonovic, S. Velankar, A. Winter, G. Wu, G. D. Bader, G. Cesareni, I. M. Donaldson, D. Eisenberg, G. J. Kleywegt, J. Overington, S. Ricard-Blum, M. Tyers, M. Albrecht, and H. Hermjakob (June 2011). "PSICQUIC and PSISCORE: accessing and scoring molecular interactions". In: *Nat. Methods* 8.7, pp. 528–529.

Archetti, D., S. Ingala, V. Venkatraghavan, V. Wottschel, A. L. Young, M. Bellio, E. E. Bron, S. Klein, F. Barkhof, D. C. Alexander, N. P. Oxtoby, G. B. Frisoni, and A. Redolfi (July 2019). "Multi-study validation of data-driven disease progression models to characterize evolution of biomarkers in Alzheimer's disease". In: *NeuroImage: Clinical* 24, p. 101954. DOI: 10.1016/j.nicl.2019.101954.

Arroyo, J. D., J. R. Chevillet, E. M. Kroh, I. K. Ruf, C. C. Pritchard, D. F. Gibson, P. S. Mitchell, C. F. Bennett, E. L. Pogosova-Agadjanyan, D. L. Stirewalt, J. F. Tait, and M. Tewari (Mar. 2011). "Argonaute2 complexes carry a population of circulating microRNAs independent of vesicles in human plasma". In: *Proc. Natl. Acad. Sci. U. S. A.* 108.12, pp. 5003-5008.

Ashburner, M, C. A. Ball, J. A. Blake, D Botstein, H Butler, J. M. Cherry, A. P. Davis, K Dolinski, S. S. Dwight, J. T. Eppig, M. A. Harris, D. P. Hill, L Issel-Tarver, A Kasarskis, S Lewis, J. C. Matese, J. E. Richardson, M Ringwald, G. M. Rubin, and
G Sherlock (May 2000). "Gene ontology: tool for the unification of biology. The Gene Ontology Consortium". In: *Nat. Genet.* 25.1, pp. 25-29.

Baig, H., P. Fontanarrosa, V. Kulkarni, J. A. McLaughlin, P. Vaidyanathan, B. Bartley, J. Beal, M. Crowther, T. E. Gorochowski, R. Grünberg, G. Misirli, J. Scott-Brown, E. Oberortner, A. Wipat, and C. J. Myers (June 2020). "Synthetic biology open language (SBOL) version 3.0.0". In: *J. Integr. Bioinform.* 17.2-3.

Baker, M. (May 2016). "1,500 scientists lift the lid on reproducibility". In: *Nature* 533.7604, pp. 452–454.

Barba, L. A. (2018). *Terminologies for Reproducible Research*. DOI: 10.48550/ARXIV.1802.03311.

Bauer-Mehren, A., L. I. Furlong, and F. Sanz (2009). "Pathway databases and tools for their exploitation: benefits, current limitations and challenges". In: *Mol Syst Biol* 5, p. 290.

Belcour, A., J. Girard, M. Aite, L. Delage, C. Trottier, C. Marteau, C. Leroux, S. M. Dittami, P. Sauleau, E. Corre, J. Nicolas, C. Boyen, C. Leblanc, J. Collén, A. Siegel, and G. V. Markov (Feb. 2020). "Inferring biochemical reactions and metabolite structures to understand metabolic pathway drift". In: *iScience* 23.2, p. 100849.

Blake, J. A. and C. J. Bult (2006). "Beyond the data deluge: Data integration and bio-ontologies". In: *Journal of Biomedical Informatics* 39.3, pp. 314–320. DOI: https://doi.org/10.1016/j.jbi.2006.01.003.

Blavy, P., F. Gondret, S. Lagarrigue, J. van Milgen, and A. Siegel (Mar. 2014). "Using a large-scale knowledge database on reactions and regulations to propose key upstream regulators of various sets of molecules participating in cell metabolism". In: *BMC Syst. Biol.* 8.1, p. 32.

Bourgon, R., R. Gentleman, and W. Huber (May 2010). "Independent filtering increases detection power for high-throughput experiments". In: *Proc. Natl. Acad. Sci. U. S. A.* 107.21, pp. 9546–9551.

Braun, P., M. Tasan, M. Dreze, M. Barrios-Rodiles, I. Lemmens, H. Yu, J. M. Sahalie, R. R. Murray, L. Roncari, A.-S. de Smet, K. Venkatesan, J.-F. Rual, J. Vandenh aute, M. E. Cusick, T. Pawson, D. E. Hill, J. Tavernier, J. L. Wrana, F. P. Roth, and M. Vi-dal (Jan. 2009). "An experimentally derived confidence score for binary protein-protein interactions". In: *Nat. Methods* 6.1, pp. 91-97.

Brennan, S., M. Keon, B. Liu, Z. Su, and N. K. Saksena (Nov. 2019). "Panoramic Visualization of Circulating MicroRNAs Across Neurodegenerative Diseases in Humans". In: *Molecular Neurobiology* 56.11, pp. 7380–7407. DOI: 10.1007/s12035-019-1615-1.

Buchka, S., A. Hapfelmeier, P. P. Gardner, R. Wilson, and A. L. Boulesteix (May 2021). "On the optimistic performance evaluation of newly introduced bioinformatic methods". In: *Genome Biol* 22.1, p. 152.

Cannata, N., E. Merelli, and R. B. Altman (2005). "Time to organize the bioinformatics resourceome". In: *PLoS Comput Biol* 1.7, e76.

Chadebec, C., E. Thibeau-Sutre, N. Burgos, and S. Allassonnière (Apr. 2021). "Data Augmentation in High Dimensional Low Sample Size Setting Using a Geometry-Based Variational Autoencoder". In: *arXiv:2105.00026 [cs, stat]*.

Chibon, F. (May 2013). "Cancer gene expression signatures - the rise and fall?" In: *Eur. J. Cancer* 49.8, pp. 2000–2009.

Chindelevitch, L., P.-R. Loh, A. Enayetallah, B. Berger, and D. Ziemek (Feb. 2012). "Assessing statistical significance in causal graphs". In: *BMC Bioinformatics* 13.1, p. 35.
Claerbout J, K. M. (1992). "Electronic documents give reproducible research a new meaning". In: *Proceedings of the 62nd Annual International Meeting of the Society of Exploration Geophysics*, pp. 601-604.

Clerx, M., M. T. Cooling, J. Cooper, A. Garny, K. Moyle, D. P. Nickerson, P. M. F. Nielsen, and H. Sorby (July 2020). "CellML 2.0". In: *J. Integr. Bioinform.* 17.2-3.

Community, T. T. W. (July 2022). *The Turing Way: A handbook for reproducible, ethical and collaborative research*. Version 1.0.2. DOI: 10.5281/zenodo.6909298.

Csardi, G. and T. Nepusz (2006). "The igraph software package for complex network research". In: *InterJournal Complex Systems*, p. 1695.

De Felice, B., A. Annunziata, G. Fiorentino, M. Borra, E. Biffali, C. Coppola, R. Cotrufo, J. Brettschneider, M. L. Giordana, T. Dalmay, G. Wheeler, and R. D'Alessandro (Oct. 2014). "miR-338-3p is over-expressed in blood, CFS, serum and spinal cord from sporadic amyotrophic lateral sclerosis patients". In: *neurogenetics* 15.4, pp. 243-253. DOI: 10.1007/s10048-014-0420-2.

Dekker, I., M. M. Schoonheim, V. Venkatraghavan, A. J. Eijlers, I. Brouwer, E. E. Bron, S. Klein, M. P. Wattjes, A. M. Wink, J. J. Geurts, B. M. Uitdehaag, N. P. Oxtoby, D. C. Alexander, H. Vrenken, J. Killestein, F. Barkhof, and V. Wottschel (Dec. 2020). "The sequence of structural, functional and cognitive changes in multiple sclerosis". In: *NeuroImage: Clinical* 29, p. 102550. DOI: 10.1016/j.nicl.2020.102550.

Delgado, F. M. and F. Gómez-Vela (Apr. 2019). "Computational methods for Gene Regulatory Networks reconstruction and analysis: A review". In: *Artif. Intell. Med.* 95, pp. 133-145.

Demir, E., O. Babur, I. Rodchenkov, B. A. Aksoy, K. I. Fukuda, B. Gross, O. S. Sümer, G. D. Bader, and C. Sander (2013). "Using biological pathway data with pathways". In: *PLoS computational biology* 9.9, e1003194.

Demir, E., M. P. Cary, S. Paley, K. Fukuda, C. Lemer, I. Vastrik, G. Wu, P. D'Eustachio, C. Schaefer, J. Luciano, F. Schacherer, I. Martinez-Flores, Z. Hu, V. Jimenez-Jacinto, G. Joshi-Tope, K. Kandasamy, A. C. Lopez-Fuentes, H. Mi, E. Pichler, I. Rodchenkov, A. Splendiani, S. Tkachev, J. Zucker, G. Gopinath, H. Rajasimha, R. Ramakrishnan, I. Shah, M. Syed, N. Anwar, O. Babur, M. Blinov, E. Brauner, D. Corwin, S. Donaldson, F. Gibbons, R. Goldberg, P. Hornbeck, A. Luna, P. Murray-Rust, E. Neumann, O. Ruebenacker, M. Samwald, M. van Iersel, S. Wimalaratne, K. Allen, B. Braun, M. Whirl-Carrillo, K.-H. Cheung, K. Dahlquist, A. Finney, M. Gillespie, E. Glass, L. Gong, R. Haw, M. Honig, O. Hubaut, D. Kane, S. Krupa, M. Kutmon, J. Leonard, D. Marks, D. Merberg, V. Petri, A. Pico, D. Ravenscroft, L. Ren, N. Shah, M. Sunshine, R. Tang, R. Whaley, S. Letovksy, K. H. Buetow, A. Rzhetsky, V. Schachter, B. S. Sobral, U. Dogrusoz, S. McWeeney, M. Aladjem, E. Birney, J. Collado-Vides, S. Goto, M. Hucka, N. Le Novère, N. Maltsev, A. Pandey, P. Thomas, E. Wingender, P. D. Karp, C. Sander, and G. D. Bader (Sept. 2010a). "The BioPAX community standard for pathway data sharing". In: *Nat. Biotechnol.* 28.9, pp. 935-942.

Demir, E., M. P. Cary, S. Paley, K. Fukuda, C. Lemer, I. Vastrik, G. Wu, P. D'Eustachio, C. a. Schaefer, F. Schacherer, I. Martinez-Flores, Z. Hu, V. Jimenez-Jacinto, G. Joshi-Tope, K. Kandasamy, A. C. Lopez-Fuentes, E. Mi Huchler, I. Rodchenkov, A. Splendiani, S. Tkachev, J. Zucker, G. Gopinath, H. Rajasimha, R. Ramakrishnan, I. Shah, N. Syeand Anwar, O. Babur, M. Blinov, E. Brauner, D. Corwin, S. Donaldson, F. Gibbons, R. Goldberg, P. Hornbeck, P. Luna Augustiy-Rust, E. Neumann, O. Ruebenacker, O. Reubenacker, M. Samwald, M. van Iersel, S. Wimalaratne, K. Allen, M. Braun Burk andillo, K.-H. Cheung, K. Dahlquist, A. Finney, M. Gillespie, E. Glass, L. Gong, R. Haw, M. Honig, O. Hubaut, S. Kand
Krupa, M. Kutmon, J. Leonard, D. Marks, D. Merberg, V. Petri, A. Pico, D. Raven-
scroft, L. Ren, N. Shah, Sugot, R. Tang, R. Whaley, S. Letovksy, K. H. Buetow, A.
Rzhetsky, V. Schachter, B. S. Sobral, U. Dogrusoz, S. a. M. McWeeney, E. Birney,
J. Collado-Vides, S. Goto, M. Hucka, N. Le Novère, N. Maltsev, A. Pandey, P.
Thomas, E. P. D. Wingender, C. Sander, and G. D. Bader (2010b). "The BioPAX
community standard for pathway data sharing". In: *Nature biotechnology* 28.9,
pp. 935-942.

Denk, J., F. Oberhauser, J. Kornhuber, J. Wiltfang, K. Fassbender, M. L. Schroeter, A. E. Volk, J. Diehl-Schmid, J. Prudlo, A. Danek, B. Landwehrmeyer, M. Lauer, M. Otto, H. Jahn, and FTLDc study group (2018). "Specific serum and CSF microRNA profiles distinguish sporadic behavioural variant of frontotemporal dementia compared with Alzheimer patients and cognitively healthy controls". In: *PloS One* 13.5, e0197329. DOI: 10.1371/journal.pone.0197329.

Diehl, A. G. and A. P. Boyle (Apr. 2016). "Deciphering ENCODE". In: *Trends Genet.* 32.4, pp. 238-249.

Dobrowolny, G., J. Martone, E. Lepore, I. Casola, A. Petrucci, M. Inghilleri, M. Mor-
lando, A. Colantoni, B. M. Scicchitano, A. Calvo, G. Bisogni, A. Chiò, M. Sabatelli,
I. Bozzoni, and A. Musarò (Jan. 2021). "A longitudinal study defined circulating
microRNAs as reliable biomarkers for disease prognosis and progression in ALS
human patients". In: *Cell Death Discovery* 7, p. 4. DOI: 10.1038/s41420-020-00397-6.

Ende, E. L. van der, E. E. Bron, J. M. Poos, L. C. Jiskoot, J. L. Panman, J. M. Papma, et al. (Oct. 2021). "A data-driven disease progression model of fluid biomarkers in genetic frontotemporal dementia". In: *Brain: A Journal of Neurology*, awab382. DOI: 10.1093/brain/awab382.

Eshaghi, A., R. V. Marinescu, A. L. Young, N. C. Firth, F. Prados, M Jorge Cardoso, C. Tur, F. De Angelis, N. Cawley, W. J. Brownlee, N. De Stefano, M Laura Stro-millo, M. Battaglini, S. Ruggieri, C. Gasperini, et al. (June 2018). "Progression of regional grey matter atrophy in multiple sclerosis". In: *Brain* 141.6, pp. 1665-1677. DOI: 10.1093/brain/awy088.

Evans, R., M. O'Neill, A. Pritzel, N. Antropova, A. Senior, T. Green, A. Žídek, R. Bates, S. Blackwell, J. Yim, O. Ronneberger, S. Bodenstein, M. Zielinski, A. Bridgland, A. Potapenko, A. Cowie, K. Tunyasuvunakool, R. Jain, E. Clancy, P. Kohli, J. Jumper, and D. Hassabis (2022). "Protein complex prediction with AlphaFold-Multimer". In: bioRxiv. DOI: 10.1101/2021.10.04.463034. eprint: https://www.biorxiv.org/content/early/2022/03/10/2021.10.04.463034.full.pdf.

Fabregat, A., K. Sidiropoulos, P. Garapati, M. Gillespie, K. Hausmann, R. Haw, B. Jassal, S. Jupe, F. Korninger, S. McKay, L. Matthews, B. May, M. Milacic, K. Rothfels, V. Shamovsky, M. Webber, J. Weiser, M. Williams, G. Wu, L. Stein, H. Hermjakob, and P. D'Eustachio (Jan. 2016). "The reactome pathway knowledgebase". In: *Nucleic Acids Res.* 44.D1, pp. D481-7.

Filangi, O., Y. Beausse, A. Assi, L. Legrand, J.-M. Larré, V. Martin, O. Collin, C. Caron, H. Leroy, and D. Allouche (Aug. 2008). "BioMAJ: a flexible framework for data-banks synchronization and processing". In: *Bioinformatics* 24.16, pp. 1823-1825.

Firth, N. C., S. Primativo, E. Brotherhood, A. L. Young, K. X. Yong, S. J. Crutch, D. C. Alexander, and N. P. Oxtoby (July 2020). "Sequences of cognitive decline in typical Alzheimer's disease and posterior cortical atrophy estimated using a novel event-based model of disease progression". In: *Alzheimer's & Dementia* 16.7, pp. 965-973. DOI: 10.1002/alz.12083.

Fonteijn, H. M., M. Modat, M. J. Clarkson, J. Barnes, M. Lehmann, N. Z. Hobbs, R. I. Scahill, S. J. Tabrizi, S. Ourselin, N. C. Fox, and D. C. Alexander (Apr. 2012).
"An event-based model for disease progression and its application in familial Alzheimer's disease and Huntington's disease". In: *NeuroImage* 60.3, pp. 1880–1889. DOI: 10.1016/j.neuroimage.2012.01.062.

Freischmidt, A., A. Goswami, K. Limm, V. L. Zimyanin, M. Demestre, H. Glaß, K. Holzmann, A. M. Helferich, S. J. Brockmann, P. Tripathi, A. Yamoah, I. Poser, P. J. Oefner, T. M. Böckers, E. Aronica, A. C. Ludolph, P. M. Andersen, A. Hermann, J. Weis, J. Reinders, K. M. Danzer, and J. H. Weishaupt (2021). "A serum microRNA sequence reveals fragile X protein pathology in amyotrophic lateral sclerosis". In: *Brain* 144.4, pp. 1214–1229. DOI: 10.1093/brain/awab018.

Freischmidt, A., K. Müller, L. Zondler, P. Weydt, B. Mayer, C. A. F. von Arnim, A. Hübers, J. Dorst, M. Otto, K. Holzmann, A. C. Ludolph, K. M. Danzer, and J. H. Weishaupt (Sept. 2015). "Serum microRNAs in sporadic amyotrophic lateral sclerosis". In: *Neurobiology of Aging* 36.9, 2660.e15–2660.e20. DOI: 10.1016/j.neurobiolaging.2015.06.003.

Freischmidt, A., K. Müller, L. Zondler, P. Weydt, A. E. Volk, A. L. Božič, M. Walter, M. Bonin, B. Mayer, C. A. F. von Arnim, M. Otto, C. Dieterich, K. Holzmann, P. M. Andersen, A. C. Ludolph, K. M. Danzer, and J. H. Weishaupt (Nov. 2014). "Serum microRNAs in patients with genetic amyotrophic lateral sclerosis and pre-manifest mutation carriers". In: *Brain: A Journal of Neurology* 137.Pt 11, pp. 2938–2950. DOI: 10.1093/brain/awu249.

Gabel, M. C., R. J. Broad, A. L. Young, S. Abrahams, M. E. Bastin, R. A. L. Menke, A. Al-Chalabi, L. H. Goldstein, S. Tsermentseli, D. C. Alexander, M. R. Turner, P. N. Leigh, and M. Cercignani (May 2020). "Evolution of white matter damage in amyotrophic lateral sclerosis". In: *Annals of Clinical and Translational Neurology* 7.5, pp. 722–732. DOI: 10.1002/acn3.51035.

Garbarino, S., M. Lorenzi, N. P. Oxtoby, E. J. Vinke, R. V. Marinescu, A. Eshaghi, M. A. Ikram, W. J. Niessen, O. Ciccarelli, F. Barkhof, J. M. Schott, M. W. Vernooij, and D. C. Alexander (2019). "Differences in topological progression profile among neurodegenerative diseases from imaging data". In: *eLife* 8. Ed. by F. P. de Lange, e49298. DOI: 10.7554/eLife.49298.

Gavin, A.-C., M. Bösche, R. Krause, P. Grandi, M. Marzioch, A. Bauer, J. Schultz, J. M. Rick, A.-M. Michon, C.-M. Cruciat, M. Remor, C. Höfert, M. Schelder, M. Brajenovic, H. Ruffner, A. Merino, K. Klein, M. Hudak, D. Dickson, T. Rudi, V. Gnau, A. Bauch, S. Bastuck, B. Huhse, C. Leutwein, M.-A. Heurtier, R. R. Copley, A. Edelmann, E. Querfurth, V. Rybin, G. Drewes, M. Raida, T. Bouwmeester, P. Bork, B. Seraphin, B. Kuster, G. Neubauer, and G. Superti-Furga (Jan. 2002). "Functional organization of the yeast proteome by systematic analysis of protein complexes". In: *Nature* 415.6868, pp. 141–147.

Gillespie, M., B. Jassal, R. Stephan, M. Milacic, K. Rothfels, A. Senff-Ribeiro, J. Griss, C. Sevilla, L. Matthews, C. Gong, C. Deng, T. Varusai, E. Ragueneau, Y. Haider, B. May, V. Shamovsky, J. Weiser, T. Brunson, N. Sanati, L. Beckman, X. Shao, A. Fabregat, K. Sidiropoulos, J. Murillo, G. Viteri, J. Cook, S. Shorser, G. Bader, E. Demir, C. Sander, R. Haw, G. Wu, L. Stein, H. Hermjakob, and P. D'Eustachio (2021). "The reactome pathway knowledgebase 2022". In: *Nucleic acids research.* DOI: https://doi.org/10.1093/nar/gkab1028.

Gleeson, P., S. Crook, R. C. Cannon, M. L. Hines, G. O. Billings, M. Farinella, T. M. Morse, A. P. Davison, S. Ray, U. S. Bhalla, S. R. Barnes, Y. D. Dimitrova, and R. A. Silver (June 2010). "NeuroML: a language for describing data driven models of neurons and networks with a high degree of biological detail". In: *PLoS Comput. Biol.* 6.6, e1000815.
Grasso, M., P. Piscopo, A. Crestini, A. Confaloni, and M. A. Denti (2015). "Circulating microRNAs in Neurodegenerative Diseases". In: *Circulating microRNAs in Disease Diagnostics and their Potential Biological Relevance*. Ed. by P. Igaz. Experimentia Supplementum. Basel: Springer, pp. 151-169. DOI: 10.1007/978-3-0348-0955-9_7.

Grasso, M., P. Piscopo, G. Talarico, L. Ricci, A. Crestini, G. Tosto, M. Gasparini, G. Bruno, M. A. Denti, and A. Confaloni (Dec. 2019). "Plasma microRNA profiling distinguishes patients with frontotemporal dementia from healthy subjects". In: *Neurobiology of Aging* 84, 240.e1-240.e12. DOI: 10.1016/j.neurobiolaging.2019.01.024.

Hafner, M., P. Landgraf, J. Ludwig, A. Rice, T. Ojo, C. Lin, D. Holoch, C. Lim, and T. Tuschl (Jan. 2008). "Identification of microRNAs and other small regulatory RNAs using cDNA library sequencing". In: *Methods* 44.1, pp. 3-12.

Hagberg, A. A., D. A. Schult, and P. J. Swart (2008). "Exploring Network Structure, Dynamics, and Function using NetworkX". In: *Proceedings of the 7th Python in Science Conference*. Ed. by G. Varoquaux, T. Vaught, and J. Millman. Pasadena, CA USA, pp. 11-15.

Hastie, T. and W. Stuetzle (1989). "Principal Curves". In: *Journal of the American Statistical Association* 84.406, pp. 502-516. DOI: 10.1080/01621459.1989.10478797.

Hermjakob, H., L. Montecchi-Palazzi, G. Bader, J. Wojcik, L. Salwinski, A. Ceol, S. Moore, S. Orchard, U. Sarkans, C. von Mering, B. Roechert, S. Poux, E. Jung, H. Mersch, P. Kersey, M. Lappe, Y. Li, R. Zeng, D. Rana, M. Nikolski, H. Husi, C. Brun, K Shanker, S. G. N. Grant, C. Sander, P. Bork, W. Zhu, A. Pandey, A. Brazma, B. Jacq, M. Vidal, D. Sherman, P. Legrain, G. Cesareni, I. Xenarios, D. Eisenberg, B. Steipe, C. Hogue, and R. Apweiler (Feb. 2004). "The HUPO PSI's molecular interaction format-a community standard for the representation of protein interaction data". In: *Nat. Biotechnol.* 22.2, pp. 177-183.

Heroux, M. A., L. Barba, M. Parashar, V. Stodden, and M. Taufer (n.d.). "Toward a Compatible Reproducibility Taxonomy for Computational and Computing Sciences." In: (). DOI: 10.2172/1481626.

Ho, Y., A. Gruhler, A. Heilbut, G. D. Bader, L. Moore, S.-L. Adams, A. Millar, P. Taylor, K. Bennett, K. Boutilier, L. Yang, C. Wolting, I. Donaldson, S. Schandorff, J. Shewnarane, M. Vo, J. Taggart, M. Goudreault, B. Muskat, C. Alfarano, D. Dewar, Z. Lin, K. Michalickova, A. R. Willems, H. Sassi, P. A. Nielsen, K. J. Rasmussen, J. R. Andersen, L. E. Johansen, L. H. Hansen, H. Jespersen, A. Podtelejnikov, E. Nielsen, J. Crawford, V. Poulsen, B. D. Sørensen, J. Matthiesen, R. C. Hendrickson, F. Gleeson, T. Pawson, M. F. Moran, D. Durocher, M. Mann, C. W. V. Hogue, D. Figeys, and M. Tyers (Jan. 2002). "Systematic identification of protein complexes in Saccharomyces cerevisiae by mass spectrometry". In: *Nature* 415.6868, pp. 180-183.

Ioannidis, J. P. A. (Aug. 2005). "Why most published research findings are false". In: *PLoS Med.* 2.8, e124.

Ito, T, T Chiba, R Ozawa, M Yoshida, M Hattori, and Y Sakaki (Apr. 2001). "A comprehensive two-hybrid analysis to explore the yeast protein interactome". In: *Proc. Natl. Acad. Sci. U. S. A.* 98.8, pp. 4569-4574.

Jedynak, B. M., A. Lang, B. Liu, E. Katz, Y. Zhang, B. T. Wyman, D. Raunig, C. P. Jedynak, B. Caffo, and J. L. Prince (Nov. 2012). "A computational neurodegenerative disease progression score: Method and results with the Alzheimer's disease neuroimaging initiative cohort". In: *NeuroImage* 63.3, pp. 1478-1486. DOI: 10.1016/j.neuroimage.2012.07.059.
Jelizarow, M., V. Guillemot, A. Tenenhaus, K. Strimmer, and A. L. Boulesteix (2010). "Over-optimism in bioinformatics: an illustration". In: *Bioinformatics* 26.16, pp. 1990–1998.

Jiang, H. and A. M. English (July 2002). "Quantitative analysis of the yeast proteome by incorporation of isotopically labeled leucine". In: *J. Proteome Res.* 1.4, pp. 345–350.

Karimi, E., E. Geslain, A. Belcour, C. Frioux, M. Aïte, A. Siegel, E. Corre, and S. M. Dittami (May 2021). "Robustness analysis of metabolic predictions in algal microbial communities based on different annotation pipelines". In: *PeerJ* 9, e11344.

Karp, P. D., C. A. Ouzounis, C. Moore-Kochlacs, L. Goldovsky, P. Kaipa, D. Ahrén, S. Tsoka, N. Darzentas, V. Kunin, and N. López-Bigas (Oct. 2005). "Expansion of the BioCyc collection of pathway/genome databases to 160 genomes". In: *Nucleic Acids Res.* 33.19, pp. 6083–6089.

Keating, S. M., D. Waltemath, M. König, F. Zhang, A. Dräger, C. Chaouiya, F. T. Bergmann, A. Finney, C. S. Gillespie, T. Helikar, S. Hoops, R. S. Malik-Sheriff, S. L. Moodie, I. I. Moraru, C. J. Myers, A. Naldi, B. G. Olivier, S. Sahle, J. C. Schaff, L. P. Smith, M. J. Swat, D. Thieffry, L. Watanabe, D. J. Wilkinson, M. L. Blinov, K. Begley, J. R. Faeder, H. F. Gómez, T. M. Hamm, Y. Inagaki, W. Liebermeister, A. L. Lister, D. Lucio, E. Mjolsness, C. J. Proctor, K. Raman, N. Rodriguez, C. A. Shaffer, B. E. Shapiro, J. Stelling, N. Swainston, N. Tanimura, J. Wagner, M. Meier-Schellersheim, H. M. Sauro, B. Palsson, H. Bolouri, H. Kitano, A. Fushahashi, H. Hermjakob, J. C. Doyle, M. Hucka, and SBML Level 3 Community members (Aug. 2020). "SBML Level 3: an extensible format for the exchange and reuse of biological models". In: *Mol. Syst. Biol.* 16.8, e9110.

Kerrien, S., B. Aranda, L. Breuza, A. Bridge, F. Broackes-Carter, C. Chen, M. Duesbury, M. Dumousseau, M. Feuermann, U. Hinz, C. Jandrasits, R. C. Jimenez, J. Khadake, U. Mahadevan, P. Masson, I. Pedruzzi, E. Pfeiffenberger, P. Porras, A. Raghunath, B. Roechert, S. Orchard, and H. Hermjakob (2012). "The IntAct molecular interaction database in 2012". In: *Nucleic Acids Res* 40.Management issue, pp. D841–846.

Kerrien, S., S. Orchard, L. Montecchi-Palazzi, B. Aranda, A. F. Quinn, N. Vinod, G. D. Bader, I. Xenarios, J. Wojcik, D. Sherman, M. Tyers, J. J. Salama, S. Moore, A. Ceol, A. Chatr-Aryamontri, M. Oesterheld, V. Stümpflen, L. Salwinski, J. Nerothin, E. Cerami, M. E. Cusick, M. Vidal, M. Gilson, J. Armstrong, P. Woollard, C. Hogue, D. Eisenberg, G. Cesareni, R. Apweiler, and H. Hermjakob (2007). "Broadening the horizon-level 2.5 of the HUPO-PSI format for molecular interactions". In: *BMC Biol* 5, p. 44.

Kim, D., G. Pertea, C. Trapnell, H. Pimentel, R. Kelley, and S. L. Salzberg (Apr. 2013). "TopHat2: accurate alignment of transcriptomes in the presence of insertions, deletions and gene fusions". In: *Genome Biol.* 14.4, R36.

Klein, A. M., L. Mazutis, I. Akartuna, N. Tallapragada, A. Veres, V. Li, L. Peshkin, D. A. Weitz, and M. W. Kirschner (May 2015). "Droplet barcoding for single-cell transcriptomics applied to embryonic stem cells". In: *Cell* 161.5, pp. 1187–1201.

Kmetzsch, V., V. Anquetil, D. Saracino, D. Rinaldi, A. Camuzat, T. Gareau, L. Jornea, S. Forlani, P. Couratier, D. Wallon, F. Pasquier, N. Robil, P. d. l. Grange, I. Moszer, I. L. Ber, O. Colliot, and E. Becker (May 2021). "Plasma microRNA signature in presymptomatic and symptomatic subjects with C9orf72-associated frontotemporal dementia and amyotrophic lateral sclerosis". In: *Journal of Neurology, Neurosurgery & Psychiatry* 92.5, pp. 485–493. DOI: 10.1136/jnnp-2020-324647.
Kolodziejczyk, A. A., J. K. Kim, V. Svensson, J. C. Marioni, and S. A. Teichmann (May 2015). "The technology and biology of single-cell RNA sequencing". In: *Mol. Cell* 58.4, pp. 610-620.

Koval, I., A. Bône, M. Louis, T. Lartigue, S. Bottani, A. Marcoux, J. Samper-González, N. Burgos, B. Charlier, A. Bertrand, S. Epelbaum, O. Colliot, S. Allassonnière, and S. Durrleman (Apr. 2021). "AD Course Map charts Alzheimer's disease progression". In: *Scientific Reports* 11.1, p. 8020. DOI: 10.1038/s41598-021-87434-1.

Langfelder, P. and S. Horvath (Dec. 2008). "WGCNA: an R package for weighted correlation network analysis". In: *BMC Bioinformatics* 9.1, p. 559.

Le Novère, N., M. Hucka, H. Mi, S. Moodie, F. Schreiber, A. Sorokin, E. Demir, K. Wegner, M. I. Aladjem, S. M. Wimalaratne, F. T. Bergman, R. Gauges, P. Ghazal, H. Kawaji, L. Li, Y. Matsuoka, A. Villéger, S. E. Boyd, L. Calzone, M. Courtot, U. Dogrusoz, T. C. Freeman, A. Funahashi, S. Ghosh, A. Jouraku, S. Kim, F. Kolpakov, A. Luna, S. Sahle, E. Schmidt, S. Watterson, G. Wu, I. Goryanin, D. B. Kell, C. Sander, H. Sauro, J. L. Snoep, K. Kohn, and H. Kitano (Aug. 2009). "The Systems Biology Graphical Notation". In: *Nat. Biotechnol.* 27.8, pp. 735-741.

Lemmens, K., T. Dhollander, T. De Bie, P. Monsieurs, K. Engelen, B. Smets, J. Winderickx, B. De Moor, and K. Marchal (May 2006). "Inferring transcriptional modules from ChIP-chip, motif and microarray data". In: *Genome Biol.* 7.5, R37.

Leng, N., J. A. Dawson, J. A. Thomson, V. Ruotti, A. I. Rissman, B. M. G. Smits, J. D. Haag, M. N. Gould, R. M. Stewart, and C. Kendziorski (Apr. 2013). "EBSeq: an empirical Bayes hierarchical model for inference in RNA-seq experiments". In: *Bioinformatics* 29.8, pp. 1035-1043.

Li, D., M. S. Zand, T. D. Dye, M. L. Goniewicz, I. Rahman, and Z. Xie (Sept. 2022). "An evaluation of RNA-seq differential analysis methods". In: *PLoS One* 17.9, e0264246.

Li, J. and R. Tibshirani (Oct. 2013). "Finding consistent patterns: a nonparametric approach for identifying differential expression in RNA-Seq data". In: *Stat. Methods Med. Res.* 22.5, pp. 519-536.

Liu, S., Z. Wang, R. Zhu, F. Wang, Y. Cheng, and Y. Liu (Sept. 2021). "Three differential expression analysis methods for RNA sequencing: Limma, EdgeR, DESeq2". In: *J. Vis. Exp.* 175.

Lorenzi, M., M. Filippone, D. C. Alexander, and S. Ourselin (Apr. 2019). "Disease Progression Modeling and Prediction through Random Effect Gaussian Processes and Time Transformation". In: *NeuroImage* 190, pp. 56-68. DOI: 10.1016/j.neuroimage.2017.08.059.

Love, M. I., W. Huber, and S. Anders (2014a). "Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2". In: *Genome Biol.* 15.12, p. 550.

— (2014b). "Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2". In: *Genome Biol.* 15.12, p. 550.

Macosko, E. Z., A. Basu, R. Satija, J. Nemesh, K. Shekhar, M. Goldman, I. Tirosh, A. R. Bialas, N. Kamitaki, E. M. Martersteck, J. J. Trombetta, D. A. Weitz, J. R. Sanes, A. K. Shalek, A. Regev, and S. A. McCarroll (May 2015). "Highly parallel genome-wide expression profiling of individual cells using nanoliter droplets". In: *Cell* 161.5, pp. 1202-1214.

Magen, I., N. S. Yacovzada, E. Yanowski, A. Coenen-Stass, J. Grosskreutz, C.-H. Lu, L. Greensmith, A. Malaspina, P. Fratta, and E. Hornstein (Nov. 2021). "Circulating miR-181 is a prognostic biomarker for amyotrophic lateral sclerosis". In: *Nature Neuroscience* 24.11, pp. 1534-1541. DOI: 10.1038/s41593-021-00936-z.

Marinescu, R. V., A. Eshaghi, M. Lorenzi, A. L. Young, N. P. Oxtoby, S. Garbarino, S. J. Crutch, D. C. Alexander, and Alzheimer's Disease Neuroimaging Initiative
(May 2019). "DIVE: A spatiotemporal progression model of brain pathology in neurodegenerative disorders". In: *NeuroImage* 192, pp. 166-177. DOI: 10.1016/j.neuroimage.2019.02.053.

Marx, V. (June 2013). "Biology: The big challenges of big data". In: *Nature* 498.7453, pp. 255-260.

McCarthy, D. J., Y. Chen, and G. K. Smyth (May 2012). "Differential expression analysis of multifactor RNA-Seq experiments with respect to biological variation". In: *Nucleic Acids Res.* 40.10, pp. 4288-4297.

Mehdipour Ghazi, M., M. Nielsen, A. Pai, M. J. Cardoso, M. Modat, S. Ourselin, and L. Sørensen (Apr. 2019). "Training recurrent neural networks robust to incomplete data: Application to Alzheimer's disease progression modeling". In: *Medical Image Analysis* 53, pp. 39-46. DOI: 10.1016/j.media.2019.01.004.

Mehdipour Ghazi, M., M. Nielsen, A. Pai, M. Modat, M. Jorge Cardoso, S. Ourselin, and L. Sørensen (Jan. 2021). "Robust parametric modeling of Alzheimer's disease progression". In: *NeuroImage* 225, p. 117460. DOI: 10.1016/j.neuroimage.2020.117460.

Mi, H., A. Muruganujan, J. T. Casagrande, and P. D. Thomas (Aug. 2013). "Large-scale gene function analysis with the PANTHER classification system". In: *Nat. Protoc.* 8.8, pp. 1551-1566.

Mitchell, P. S., R. K. Parkin, E. M. Kroh, B. R. Fritz, S. K. Wyman, E. L. Pogosova-Agadjanyan, A. Peterson, J. Noteboom, K. C. O'Briant, A. Allen, D. W. Lin, N. Urban, C. W. Drescher, B. S. Knudsen, D. L. Stirewalt, R. Gentleman, R. L. Vessella, P. S. Nelson, D. B. Martin, and M. Tewari (July 2008). "Circulating microRNAs as stable blood-based markers for cancer detection". In: *Proc. Natl. Acad. Sci. U. S. A.* 105.30, pp. 10513-10518.

Mortazavi, A., B. A. Williams, K. McCue, L. Schaeffer, and B. Wold (July 2008). "Mapping and quantifying mammalian transcriptomes by RNA-Seq". In: *Nat. Methods* 5.7, pp. 621-628.

Nurse, P. (Sept. 2021). "Biology must generate ideas as well as data". In: *Nature* 597.7876, p. 305.

Oda, Y, K Huang, F. R. Cross, D Cowburn, and B. T. Chait (June 1999). "Accurate quantitation of protein expression and site-specific phosphorylation". In: *Proc. Natl. Acad. Sci. U. S. A.* 96.12, pp. 6591-6596.

Ong, S.-E., B. Blagoev, I. Kratchmarova, D. B. Kristensen, H. Steen, A. Pandey, and M. Mann (May 2002). "Stable isotope labeling by amino acids in cell culture, SILAC, as a simple and accurate approach to expression proteomics". In: *Mol. Cell. Proteomics* 1.5, pp. 376-386.

Orchard, S., S. Kerrien, S. Abbani, B. Aranda, J. Bhate, S. Bidwell, A. Bridge, L. Briganti, F. S. Brinkman, F. Brinkman, G. Cesareni, A. Chatr-aryamontri, E. Chautard, C. Chen, M. Dumousseau, J. Goll, R. E. Hancock, R. Hancock, L. I. Hannick, I. Jurisica, J. Khadake, D. J. Lynn, U. Mahadevan, L. Perfetto, A. Raghunath, S. Ricard-Blum, B. Roechert, L. Salwinski, V. Stümpflen, M. Tyers, P. Uetz, I. Xenarios, and H. Hermjakob (2012). "Protein interaction data curation: the International Molecular Exchange (IMEx) consortium". In: *Nat Methods* 9.4, pp. 345-350.

Oxtoby, N. P., L.-A. Leyland, L. M. Aksman, G. E. C. Thomas, E. L. Bunting, P. A. Wijeratne, A. L. Young, A. Zarkali, M. M. X. Tan, F. D. Bremner, P. A. Keane, H. R. Morris, A. E. Schrag, D. C. Alexander, and R. S. Weil (Feb. 2021). "Sequence of clinical and neurodegeneration events in Parkinson's disease progression". In: *Brain* 144.3, pp. 975-988. DOI: 10.1093/brain/awaa461.

Oxtoby, N. P., A. L. Young, D. M. Cash, T. L. S. Benzinger, A. M. Fagan, J. C. Morris, R. J. Bateman, N. C. Fox, J. M. Schott, and D. C. Alexander (May 2018).
"Data-driven models of dominantly-inherited Alzheimer's disease progression". In: *Brain* 141.5, pp. 1529-1544. DOI: 10.1093/brain/awy050.

Panman, J. L., V. Venkatraghavan, E. L. v. d. Ende, R. M. E. Steketee, L. C. Jiskoot, J. M. Poos, E. G. P. Dopper, et al. (Jan. 2021). "Modelling the cascade of biomarker changes in GRN-related frontotemporal dementia". In: *Journal of Neurology, Neurosurgery & Psychiatry*. DOI: 10.1136/jnnp-2020-323541.

Peng, R. D., F. Dominici, and S. L. Zeger (May 2006). "Reproducible epidemiologic research". In: *Am. J. Epidemiol*. 163.9, pp. 783-789.

Peri, S., J. D. Navarro, R. Amanchy, T. Z. Kristiansen, C. K. Jonnalagadda, V. Suredranath, V. Niranjan, B. Muthusamy, T. K. B. Gandhi, M. Gronborg, N. Ibarrola, N. Deshpande, K. Shanker, H. N. Shivashankar, B. P. Rashmi, M. A. Ramya, Z. Zhao, K. N. Chandrika, N. Padma, H. C. Harsha, A. J. Yatish, M. P. Kavitha, M. Menezes, D. R. Choudhury, S. Suresh, N. Ghosh, R. Saravana, S. Chandran, S. Krishna, M. Joy, S. K. Anand, V. Madavan, A. Joseph, G. W. Wong, W. P. Schiemann, S. N. Constantinescu, L. Huang, R. Khosravi-Far, H. Steen, M. Tewari, S. Ghaffari, G. C. Globe, C. V. Dang, J. G. N. Garcia, J. Pevsner, O. N. Jensen, P. Roepstorff, K. S. Deshpande, A. M. Chinnaiyan, A. Hamosh, A. Chakravarti, and A. Pandey (Oct. 2003). "Development of human protein reference database as an initial platform for approaching systems biology in humans". In: *Genome Res.* 13.10, pp. 2363-2371.

Picelli, S., Å. K. Björklund, O. R. Faridani, S. Sagasser, G. Winberg, and R. Sandberg (Nov. 2013). "Smart-seq2 for sensitive full-length transcriptome profiling in single cells". In: *Nat. Methods* 10.11, pp. 1096-1098.

Piscopo, P., M. Grasso, M. Puopolo, E. D'Acunto, G. Talarico, A. Crestini, M. Gasparini, R. Campopiano, S. Gambardella, A. E. Castellano, G. Bruno, M. A. Denti, and A. Confaloni (Jan. 2018). "Circulating miR-127-3p as a Potential Biomarker for Differential Diagnosis in Frontotemporal Dementia". In: *Journal of Alzheimer's Disease* 65.2, pp. 455-464. DOI: 10.3233/JAD-180364.

Poldrack, R. A., G. Huckins, and G. Varoquaux (May 2020). "Establishment of best practices for evidence for prediction: A review". In: *JAMA Psychiatry* 77.5, pp. 534-540.

Porras, P., E. Barrera, A. Bridge, N. Del-Toro, G. Cesareni, M. Duesbury, H. Hermjakob, M. Iannuccelli, I. Jurisica, M. Kotlyar, L. Licata, R. C. Lovering, D. J. Lynn, B. Meldal, B. Nanduri, K. Paneerselvam, S. Panni, C. Pastrello, M. Pellegrini, L. Perfetto, N. Rahimzadeh, P. Ratan, S. Ricard-Blum, L. Salwinski, G. Shirodkar, A. Shrivastava, and S. Orchard (2020). "Towards a unified open access dataset of molecular interactions". In: *Nature communications* 11.1, p. 6144. DOI: https://doi.org/10.1038/s41467-020-19942-z.

Raheja, R., K. Regev, B. C. Healy, M. A. Mazzola, V. Beynon, F. von Glehn, A. Paul, C. Diaz-Cruz, T. Gholipour, B. I. Glanz, P. Kivisakk, T. Chitnis, H. L. Weiner, J. D. Berry, and R. Gandhi (Aug. 2018). "Correlating serum microRNAs and clinical parameters in Amyotrophic lateral sclerosis". In: *Muscle & nerve* 58.2, pp. 261-269. DOI: 10.1002/mus.26106.

Rau, A., M. Gallopin, G. Celeux, and F. Jaffrézic (Sept. 2013). "Data-based filtering for replicated high-throughput transcriptome sequencing experiments". In: *Bioinformatics* 29.17, pp. 2146-2152.

Reverter, A. and E. K. F. Chan (Nov. 2008). "Combining partial correlation and an information theory approach to the reversed engineering of gene co-expression networks". In: *Bioinformatics* 24.21, pp. 2491-2497.
Robinson, M. D., D. J. McCarthy, and G. K. Smyth (Jan. 2010a). "edgeR: a Bioconductor package for differential expression analysis of digital gene expression data". In: *Bioinformatics* 26.1, pp. 139–140.

— (Jan. 2010b). "edgeR: a Bioconductor package for differential expression analysis of digital gene expression data". In: *Bioinformatics* 26.1, pp. 139–140.

Robinson, M. D. and A. Oshlack (Mar. 2010). "A scaling normalization method for differential expression analysis of RNA-seq data". In: *Genome Biol*. 11.3, R25.

Rodchenkov, I., E. Demir, C. Sander, and G. D. Bader (2013). "The BioPAX Validator". In: *Bioinformatics* 29.20, pp. 2659–2660.

Rodchenkov, I., O. Babur, A. Luna, B. A. Aksoy, J. V. Wong, D. Fong, M. Franz, M. C. Siper, M. Cheung, M. Wrana, H. Mistry, L. Mosier, J. Dlin, Q. Wen, C. O'Callaghan, W. Li, G. Elder, P. T. Smith, C. Dallago, E. Cerami, B. Gross, U. Dogrusoz, E. Demir, G. D. Bader, and C. Sander (Jan. 2020a). "Pathway Commons 2019 Update: integration, analysis and exploration of pathway data". In: *Nucleic Acids Res.* 48.D1, pp. D489–D497.

— (Jan. 2020b). "Pathway Commons 2019 Update: integration, analysis and exploration of pathway data". In: *Nucleic Acids Res.* 48.D1, pp. D489–D497.

Romero, P., J. Wagg, M. L. Green, D. Kaiser, M. Krummenacker, and P. D. Karp (2005). "Computational prediction of human metabolic pathways from the complete human genome". In: *Genome Biol.* 6.1, R2.

Schiratti, J.-B., S. Allassonnière, O. Colliot, and S. Durrleman (2017). "A Bayesian Mixed-Effects Model to Learn Trajectories of Changes from Repeated Manifold-Valued Observations". In: *Journal of Machine Learning Research* 18.133, pp. 1–33.

Schmidt, D., M. D. Wilson, C. Spyrou, G. D. Brown, J. Hadfield, and D. T. Odom (July 2009). "ChIP-seq: using high-throughput sequencing to discover protein-DNA interactions". In: *Methods* 48.3, pp. 240–248.

Schreiber, F., B. Sommer, G. D. Bader, P. Gleeson, M. Golebiewski, M. Hucka, S. M. Keating, M. König, C. Myers, D. Nickerson, and D. Waltemath (July 2019). "Specifications of standards in systems and synthetic biology: Status and developments in 2019". In: *J. Integr. Bioinform.* 16.2.

Schurch, N. J., P. Schofield, M. Gierliński, C. Cole, A. Sherstnev, V. Singh, N. Wrobel, K. Gharbi, G. G. Simpson, T. Owen-Hughes, M. Blaxter, and G. J. Barton (Oct. 2016). "Erratum: How many biological replicates are needed in an RNA-seq experiment and which differential expression tool should you use?" In: *RNA* 22.10, p. 1641.

Seyednasrollah, F., A. Laiho, and L. L. Elo (Jan. 2015). "Comparison of software packages for detecting differential expression in RNA-seq studies". In: *Brief. Bioinform.* 16.1, pp. 59–70.

Sheinerman, K. S., J. B. Toledo, V. G. Tsivinsky, D. Irwin, M. Grossman, D. Weintraub, H. I. Hurtig, A. Chen-Plotkin, D. A. Wolk, L. F. McCluskey, L. B. Elman, J. Q. Trojanowski, and S. R. Umansky (Nov. 2017). "Circulating brain-enriched microRNAs as novel biomarkers for detection and differentiation of neurodegenerative diseases". In: *Alzheimer's Research & Therapy* 9.1, p. 89. DOI: 10.1186/s13195-017-0316-0.

Singh, A., C. P. Shannon, B. Gautier, F. Rohart, M. Vacher, S. J. Tebbutt, and K.-A. Lê Cao (Sept. 2019). "DIABLO: an integrative approach for identifying key molecular drivers from multi-omics assays". In: *Bioinformatics* 35.17, pp. 3055–3062.

Sivade Dumousseau, M., D. Alonso-López, M. Ammari, G. Bradley, N. H. Campbell, A. Ceol, G. Cesareni, C. Combe, J. De Las Rivas, N. Del-Toro, J. Heimbach, H. Hermjakob, I. Jurisica, M. Koch, L. Licata, R. C. Lovering, D. J. Lynn, B. H. M.
Meldal, G. Micklem, S. Panni, P. Porras, S. Ricard-Blum, B. Roechert, L. Salwin-ski, A. Shrivastava, J. Sullivan, N. Thierry-Mieg, Y. Yehudi, K. Van Roey, and S. Orchard (Apr. 2018). "Encompassing new use cases - level 3.0 of the HUPO-PSI format for molecular interactions". In: *BMC Bioinformatics* 19.1, p. 134.

Soliman, R., N. O. Mousa, H. R. Rashed, R. R. Moustafa, N. Hamdi, A. Osman, and N. Fahmy (Sept. 2021). "Assessment of diagnostic potential of some circulating microRNAs in Amyotrophic Lateral Sclerosis Patients, an Egyptian study". In: *Clinical Neurology and Neurosurgery* 208, p. 106883. DOI: 10.1016/j.clineuro.2021.106883.

Soneson, C. and M. Delorenzi (Mar. 2013). "A comparison of methods for differential expression analysis of RNA-seq data". In: *BMC Bioinformatics* 14.1, p. 91.

Sreedharan, J., I. P. Blair, V. B. Tripathi, X. Hu, C. Vance, B. Rogelj, S. Ackerley, J. C. Durnall, K. L. Williams, E. Buratti, F. Baralle, J. de Belleroche, J. D. Mitchell, P. N. Leigh, A. Al-Chalabi, C. C. Miller, G. Nicholson, and C. E. Shaw (Mar. 2008). "TDP-43 mutations in familial and sporadic amyotrophic lateral sclerosis". In: *Science* 319.5870, pp. 1668-1672.

Stephens, Z. D., S. Y. Lee, F. Faghri, R. H. Campbell, C. Zhai, M. J. Efron, R. Iyer, M. C. Schatz, S. Sinha, and G. E. Robinson (July 2015). "Big Data: Astronomical or genomical?" In: *PLoS Biol*. 13.7, e1002195.

Svensson, V., R. Vento-Tormo, and S. A. Teichmann (Apr. 2018). "Exponential scaling of single-cell RNA-seq in the past decade". In: *Nat. Protoc*. 13.4, pp. 599-604.

Szklarczyk, D., A. L. Gable, K. C. Nastou, D. Lyon, R. Kirsch, S. Pyysalo, N. T. Doncheva, M. Legeay, T. Fang, P. Bork, L. J. Jensen, and C. von Mering (Jan. 2021). "The STRING database in 2021: customizable protein-protein networks, and functional characterization of user-uploaded gene/measurement sets". In: *Nucleic Acids Res*. 49.D1, pp. D605-D612.

Takahashi, I., Y. Hama, M. Matsushima, M. Hirotani, T. Kano, H. Hohzen, I. Yabe, J. Utsumi, and H. Sasaki (Oct. 2015). "Identification of plasma microRNAs as a biomarker of sporadic Amyotrophic Lateral Sclerosis". In: *Molecular Brain* 8.1, p. 67. DOI: 10.1186/s13041-015-0161-7.

Tarazona, S., F. García-Alcalde, J. Dopazo, A. Ferrer, and A. Conesa (Dec. 2011). "Differential expression in RNA-seq: a matter of depth". In: *Genome Res*. 21.12, pp. 2213-2223.

Tasca, E., V. Pegoraro, A. Merico, and C. Angelini (Jan. 2016). "Circulating microRNAs as biomarkers of muscle differentiation and atrophy in ALS". In: *Clinical Neuropathology* 35.01, pp. 22-30. DOI: 10.5414/NP300889.

The Gene Ontology Consortium (Jan. 2019). "The Gene Ontology Resource: 20 years and still GOing strong". In: *Nucleic Acids Res*. 47.D1, pp. D330-D338.

Thomas, P. D., D. Ebert, A. Muruganujan, T. Mushayahama, L.-P. Albou, and H. Mi (Jan. 2022). "PANTHER: Making genome-scale phylogenetics accessible to all". In: *Protein Sci*. 31.1, pp. 8-22.

Thomas, P. D., A. Kejariwal, M. J. Campbell, H. Mi, K. Diemer, N. Guo, I. Ladunga, B. Ulitsky-Lazareva, A. Muruganujan, S. Rabkin, J. A. Vandergriff, and O. Doremieux (Jan. 2003). "PANTHER: a browsable database of gene products organized by biological function, using curated protein family and subfamily classification". In: *Nucleic Acids Res*. 31.1, pp. 334-341.

Toro, N. del, M. Dumousseau, S. Orchard, R. C. Jimenez, E. Galeota, G. Launay, J. Goll, K. Breuer, K. Ono, L. Salwinski, and H. Hermjakob (2013). "A new reference implementation of the PSICQUIC web service". In: *Nucleic Acids Res* 41.Web Server issue, W601-606.
Trapnell, C., D. G. Hendrickson, M. Sauvageau, L. Goff, J. L. Rinn, and L. Pachter (Jan. 2013). "Differential analysis of gene regulation at transcript resolution with RNA-seq". In: *Nat. Biotechnol.* 31.1, pp. 46–53.

Uetz, P., L Giot, G Cagney, T. A. Mansfield, R. S. Judson, J. R. Knight, D Lockshon, V Narayan, M Srinivasan, P Pochart, A Qureshi-Emili, Y Li, B Godwin, D Conover, T Kalbfleisch, G Vijayadamodar, M Yang, M Johnston, S Fields, and J. M. Rothberg (Feb. 2000). "A comprehensive analysis of protein-protein interactions in Saccharomyces cerevisiae". In: *Nature* 403.6770, pp. 623–627.

Ullmann, T., S. Peschel, P. Finger, C. L. Müller, and A.-L. Boulesteix (2022). "Over-optimism in unsupervised microbiome analysis: Insights from network learning and clustering". In: *bioRxiv*. DOI: 10.1101/2022.06.24.497500. eprint: https://www.biorxiv.org/content/early/2022/06/28/2022.06.24.497500.full.pdf.

Valdeolivas, A., L. Tichit, C. Navarro, S. Perrin, G. Odelin, N. Levy, P. Cau, E. Remy, and A. Baudot (Feb. 2019). "Random walk with restart on multiplex and heterogeneous biological networks". In: *Bioinformatics* 35.3, pp. 497–505.

Venkatraghavan, V., E. E. Bron, W. J. Niessen, and S. Klein (Feb. 2019). "Disease progression timeline estimation for Alzheimer's disease using discriminative event based modeling". In: *NeuroImage* 186, pp. 518–532. DOI: 10.1016/j.neuroimage.2018.11.024.

Villaveces, J. M., R. C. Jiménez, P. Porras, N. del Toro, M. Duesbury, M. Dumousseau, S. Orchard, H. Choi, P. Ping, N. C. Zong, M. Askenazi, B. H. Habermann, and H. Hermjakob (Feb. 2015). "Merging and scoring molecular interactions utilising existing community standards: tools, use-cases and a case study". In: *Database* 2015. DOI: 10.1093/database/bau131. eprint: https://academic.oup.com/database/article-pdf/doi/10.1093/database/bau131/7298497/bau131.pdf.

Wagner, A., A. Regev, and N. Yosef (Nov. 2016). "Revealing the vectors of cellular identity with single-cell genomics". In: *Nat. Biotechnol.* 34.11, pp. 1145–1160.

Waller, R., E. F. Goodall, M. Milo, J. Cooper-Knock, M. Da Costa, E. Hobson, M. Kazoka, H. Wollff, P. R. Heath, P. J. Shaw, and J. Kirby (July 2017). "Serum miRNAs miR-206, 143-3p and 374b-5p as potential biomarkers for amyotrophic lateral sclerosis (ALS)". In: *Neurobiology of Aging* 55, pp. 123–131. DOI: 10.1016/j.neurobiolaging.2017.03.027.

Wang, Z., M. Gerstein, and M. Snyder (Jan. 2009a). "RNA-Seq: a revolutionary tool for transcriptomics". In: *Nature Reviews Genetics* 10.1, pp. 57–63. DOI: 10.1038/nrg2484.

— (Jan. 2009b). "RNA-Seq: a revolutionary tool for transcriptomics". In: *Nat. Rev. Genet.* 10.1, pp. 57–63.

Wijeratne, P. A., E. B. Johnson, S. Gregory, N. Georgiou-Karistianis, J. S. Paulsen, R. I. Scahill, S. J. Tabrizi, and D. C. Alexander (Aug. 2021). "A Multi-Study Model-Based Evaluation of the Sequence of Imaging and Clinical Biomarker Changes in Huntington's Disease". In: *Frontiers in Big Data* 4, p. 662200. DOI: 10.3389/fdata.2021.662200.

Wilkinson, M. D., M. Dumontier, I. J. J. Aalbersberg, G. Appleton, M. Axton, A. Baak, N. Blomberg, J.-W. Boiten, L. B. da Silva Santos, P. E. Bourne, J. Bouwman, A. J. Brookes, T. Clark, M. Crosas, I. Dillo, O. Dumon, S. Edmunds, C. T. Evelo, R. Finkers, A. Gonzalez-Beltran, A. J. G. Gray, P. Groth, C. Goble, J. S. Grethe, J. Heringa, P. A. C. 't Hoen, R. Hooft, T. Kuhn, R. Kok, J. Kok, S. J. Lusher, M. E. Martone, A. Mons, A. L. Packer, B. Persson, P. Rocca-Serra, M. Roos, R. van Schaik, S.-A. Sansone, E. Schultes, T. Sengstag, T. Slater, G. Strawn, M. A. Swertz, M. Thompson, J. van der Lei, E. van Mulligen, J. Velterop, A. Waagmeester, P. Wittenburg, K. Wolstencroft, J. Zhao, and B. Mons (2016). "The FAIR Guiding
Principles for scientific data management and stewardship". In: *Scientific data* 3, p. 160018.

Xu, Q., Y. Zhao, X. Zhou, J. Luan, Y. Cui, and J. Han (Feb. 2018). "Comparison of the extraction and determination of serum exosome and miRNA in serum and the detection of miR-27a-3p in serum exosome of ALS patients". In: *Intractable & Rare Diseases Research* 7.1, pp. 13–18. DOI: 10.5582/irdr.2017.01091.

Young, A. L., N. P. Oxtoby, P. Daga, D. M. Cash, N. C. Fox, S. Ourselin, J. M. Schott, D. C. Alexander, and on behalf of the Alzheimer's Disease Neuroimaging Initiative (Sept. 2014). "A data-driven model of biomarker changes in sporadic Alzheimer's disease". In: *Brain* 137.9, pp. 2564–2577. DOI: 10.1093/brain/awu176.