Graph Neural Networks in
Computational Biology(a Personal Perspective)
Petar Veličković
Computational Biology Society Seminar
Imperial College London
19 April 2021

In this talk:
Graph neural networks for biological data

What this talk is not!

What this talk is not! 🙈

For more on AlphaFold, see:https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-gr
and-challenge-in-biology

What this talk is●

Hopefully an exciting ﬁeld from many angles :)○ Molecular interactions○ Protein function prediction○ Genome assembly○ Computational neuroscience○ Electronic Health Records

What this talk is●Hopefully an exciting ﬁeld from many angles :)○ Molecular interactions○ Protein function prediction○ Genome assembly○ Computational neuroscience○ Electronic Health Records●More broadly…
○ Personal perspective on this rich, interdisciplinary ﬁeld
○ For ML audience: you can do it!
■ + a blueprint for approaching the area
○ For Bio audience: hopefully a useful computational tool(for both: interdisciplinary collaboration can work wonders!)

Let’s start at the beginning●

Born in Belgrade (Serbia) in the 1990s○ Family members worked for local representatives of “big pharma” (Merck)

Let’s start at the beginning●

Born in Belgrade (Serbia) in the 1990s○ Family members worked for local representatives of “big pharma” (Merck)●

Gradually increasing interest towards computer science – especially classical algorithms

Let’s start at the beginning●

Born in Belgrade (Serbia) in the 1990s○ Family members worked for local representatives of “big pharma” (Merck)●

Gradually increasing interest towards computer science – especially classical algorithms●Developed strong interest in biology in high school (primarily thanks to Branka Dobrković)

Let’s start at the beginning●

Born in Belgrade (Serbia) in the 1990s○ Family members worked for local representatives of “big pharma” (Merck)●

Gradually increasing interest towards computer science – especially classical algorithms●

Developed strong interest in biology in high school (primarily thanks to Branka Dobrković)●

Computer Science at Cambridge (2012--15)○ Lost nearly all contact with biology●

Reached out to Prof Pietro Liò for my ﬁnal-year project○ Realised that bioinformatics is brimming with classical algorithms○ Pietro suggested a project in machine learning, however...
○ The rest is history (i.e. this talk)Before GNNs...
●I started my PhD in 2016, with a paper classifying breast cancer●Ofﬁcially I was a “Research Assistant in Computational Biology”○ But no formal training in biology!
○ Luckily, the ﬁeld is remarkably accessible and full of interesting problems to solve○ It was very helpful to talk to domain experts and understand the “burning questions”●

Fruitful collaborations lead to Parapred (Bioinformatics) and ChronoMID (PLOS ONE)○ Carefully crafted machine learning solutions to problems posed by domain experts

Before GNNs...
●I started my PhD in 2016, with a paper classifying breast cancer●Ofﬁcially I was a “Research Assistant in Computational Biology”○ But no formal training in biology!
○ Luckily, the ﬁeld is remarkably accessible and full of interesting problems to solve○ It was very helpful to talk to domain experts and understand the “burning questions”●

Fruitful collaborations lead to Parapred (Bioinformatics) and ChronoMID (PLOS ONE)○ Carefully crafted machine learning solutions to problems posed by domain experts●“Game changing” moment in 2017, when I discovered graph representation learning○ Why should you care?

Molecules are graphs!
●A very natural way to represent molecules is as a graph○ Atoms as nodes, bonds as edges○ Features such as atom type, charge, bond type...

GNNs for molecule classification●Interesting task to predict is, for example, whether the molecule is a potent drug.
○ Can do binary classiﬁcation on whether the drug will inhibit certain bacteria. (E.coli)○ Train on a curated dataset for compounds where response is known.

GNNMolecule

Inhibits E.coli?

Follow-up study●Once trained, the model can be applied to any molecule.
○ Execute on a large dataset of known candidate molecules.
○ Select the ~top-100 candidates from your GNN model.
○ Have chemists thoroughly investigate those (after some additional ﬁltering).

●

Discover a previously overlooked compound that is a highly potent antibiotic!

Halicin...Achieve wide acclaim!
Arguably the most popularised success story of graph neural networks to date!

(Stokes et al., Cell’20)

...Achieve wide acclaim!
Arguably the most popularised success story of graph neural networks to date!

(Stokes et al., Cell’20)

...Achieve wide acclaim!
Arguably the most popularised success story of graph neural networks to date!

(Stokes et al., Cell’20)

...Achieve wide acclaim!
Arguably the most popularised success story of graph neural networks to date!

(Stokes et al., Cell’20)

GNNs are a very hot research topic

GNNs are currently experiencing their“ImageNet” moment

Rich ecosystem of libraries

github.com/rusty1s/pytorch_geometricgraphneural.network

dgl.aigithub.com/deepmind/graph_nets

github.com/deepmind/jraphRich ecosystem of datasets

ogb.stanford.edu

https://pytorch-geometric.readthedocs.
io/en/latest/modules/datasets.html

github.com/graphdeeplearning/benchmarking-gnns

graphlearning.io

How to process the graph?

What’s in a GNN layer?
●

We construct useful functions over graphs, f, by shared application of a local
permutation-invariant function g(xi, XNi).
○ We often refer to f as “GNN layer”, g as “diffusion”, “propagation”, “message passing”●

We will take a quick look at ways in which we can actually concretely deﬁne g.
○ Very intense area of research!

●Fortunately, almost all proposed layers can be classiﬁed as one of three spatial “ﬂavours”.

The three “flavours” of GNN layers

Convolutional GNN●Features of neighbours aggregated with ﬁxed weights, cij●Usually, the weights depend directly on A.
○ ChebyNet (Defferrard et al., NeurIPS’16)○ GCN (Kipf & Welling, ICLR’17)○ SGC (Wu et al., ICML’19)●

Useful for homophilous graphs and scaling up○ When edges encode label similarityAttentional GNN●Features of neighbours aggregated with implicit weights (via attention)●Attention weight computed as ɑij = a(xi, xj)○ MoNet (Monti et al., CVPR’17)○ GAT (Veličković et al., ICLR’18)○ GaAN (Zhang et al., UAI’18)●

Useful as “middle ground” w.r.t. capacity and scale○ Edges need not encode homophily○ But still compute scalar value in each edge

Message-passing GNN●Compute arbitrary vectors (“messages”) to be sent across edges●Messages computed as mij = 𝜓(xi, xj)○ Interaction Networks (Battaglia et al., NeurIPS’16)○ MPNN (Gilmer et al., ICML’17)○ GraphNets (Battaglia et al., 2018)●Most generic GNN layer○ May have scalability or learnability issues○ Ideal for computational chemistry, reasoning and simulation

How to use GNNs?

How to use GNNs?

How to use GNNs?

How to use GNNs?

How to use GNNs?

If you’d like to know more
For (substantially!) more context, I recently gave a talk on theoretical GNN foundations:
https://www.youtube.com/watch?v=uF53xsT7mjc

...Back to the past 🕸●In 2017, as part of my Mila internship we proposed Graph Attention Networks (GATs)○ One of the ﬁrst prominent examples of attentional GNN○ They remain a popular model to this day●

It was only loosely clear that models like these could beneﬁt my biological projects○ We set out to ﬁnd out exactly how...

CNNs* for Mesh-based Parcellationof the Cerebral Cortex
Guillem Cucurull, Konrad Wagstyl, Arantxa Casanova, Petar Veličković,
Estrid Jakobsen, Michal Drozdzal, Adriana Romero, Alan Evans and Yoshua Bengio

Cortex parcellation●Different areas of the cerebral cortex are
involved in different cognitive processes○ Visual processing○ Language comprehension●

Mapping these areas helps us understand how
the cortex is organised●

Our graph attention network paper was, in fact,
built for this very purpose :)●We focus on regions 44 and 45 of Broca’s area:

What is a cortical mesh?
●

Common coordinate system●

Can represent multiple modalities and features●

Can be used to coregister cortical surfaces
between different individuals●

We can run a GNN over the nodes in the mesh!
○ Classify nodes as “44”, “45”, or “background”.

Quantitative results

Quantitative results

Quantitative results

Quantitative results+ Node positional features!

Qualitative results

With hindsight...
●Meshes come with a lot of useful geometry○ Evident by utility of positional features○ (Vanilla) GNNs would discard that information●

We now have a wealth of architectures that are specialising for the mesh domain!
○ Geodesic CNN (Masci et al.)
○ MoNet (Monti et al.)○ Gauge Equivariant Mesh CNN (de Haan et al.)●All of the above would make great choices for processing the brain mesh!
○ Perhaps an interesting future project? 👀○ Reach out to me if you’re curious!

...Back to the past 🧠●This project proved to me the untapped utility that GNNs can have in biological problems○ We applied the GCN and GAT models pretty much out-of-the-box!

●

Now was the time to revisit my earlier collaboration (Parapred) under this lens.

Attentive Cross-modal
Paratope Prediction
Andreea Deac, Petar Veličković and Pietro Sormanni

Motivation for antibody design●Antibodies are○ Y-shaped proteins○ a critical part of our immune system●

They neutralise pathogenic bacteria and viruses bytagging the antigen in a “lock and key” system.

●

Designing our own arbitrary antibodies would be a big
step towards personalised medicine.

●(You’ve probably heard a whole lot about antibodies
and antigens in the past year...)Towards personalised medicine●

Generating an antibody requires ﬁrst predicting the speciﬁc amino acids (the paratope)
which participate in the neutralisation of the antigen.

●

Input: a sequence of (one-hot encoded) antibody amino acids.
(+ a sequence of (one-hot encoded) antigen amino acids)●Output: probability for each amino acid to participate in the binding with the antigen.

Paratope prediction

Parapred and Fast-Parapred architectureParatope prediction (+ antigen)Fast-Parapred

AG-Fast-Parapred

AG-Fast-Parapred~ GAT over fully-connected antibody/antigen graph!

Quantitative results

Qualitative results

The model learns the antibody/antigen geometry without being given any positional information!

...Back to the past 🦠●Now it was apparent that stitching GNNs into protein-protein interaction made sense!

●Could we explore some other cases of molecular interaction?

Drug-Drug Adverse EffectPrediction with Graph Co-Attention
Andreea Deac, Yu-Hsiang Huang, Petar Veličković, Pietro Liò and Jian Tang

Drug use is increasing

Prescription Drug Use>5 drugs

2000

2011

51%

59%

8.2%

15%

Polypharmacy●

Polypharmacy is the concurrent use of multiple medications by a patient.

●It is necessary for chronic, complex or multiple conditions and most of the increase in costcomes from treating these.

●“Hulk & Iron Man” analogy: drugs correspond to ‘heroes’, but putting them together candestroy the surrounding city!

Adverse side-effects●Side effects affecting 15% of the population, treatment costs exceeding $177 billion/year●

Some found in Phase IV of clinical trials●

But plenty are undiscovered when the drugs are put on the market

Related work●

Most models predict if a side-effect exists ornot (using drug-drug similarity: chemicalsubstructures, individual drug side effects,
interaction proﬁle ﬁngerprints)●

Others model the interactions between pairs ofdrugs, pairs of proteins and drug-protein pairs
to predict “missing” links between them.

●We, instead, represents molecules as graphs!

* Modeling polypharmacy side effects with graph convolutional networks, Žitnik et al, 2018

DDI - Tasks

Graph co-attention

The (MH)CADDI ArchitectureVariants consideredQuantitative results

...Back to the past 💊●It was ~at this point I graduated from my PhD, and joined DeepMind●

Gradually oriented back towards classical algorithms, and away from biology○ Luckily, biology is packed with interesting classical algorithms :)●The following three works (time permitting) represent a medley of biological approaches I
was involved in during this time.
○ Two of these opportunities came not far from home :)○ The third one was years in the making!

Hierarchical Protein Function
Prediction with Tail-GNNs
Stefan Spalević, Petar Veličković, Jovana Kovačević and Mladen Nikolić

Protein function prediction●Detecting mechanisms of action for proteins is a highly relevant task!

●It is also an area ripe with graphs!
○ Protein itself can be represented as a graph (if known structure; Gligorijević et al.)○ Protein-protein interaction networks are graphs (standard PPI benchmark for GNNs)○ In this particular domain, a graph comes up in one more place...

Protein function prediction●The label space of functions is itself a graph! (gene ontology)●Requires a GNN in the label space○ Our literature survey suggested no proposals like this!
○ Once again, a biological problem motivates a core architecture

Tail-GNNQuantitative results●With the right aggregator choice + spectral features, yields signiﬁcant beneﬁts!

A Step Towards
Neural Genome Assembly
Lovro Vrček, Petar Veličković and Mile ŠikićGenome assemblyGenome assemblyGenome assemblyGenome assembly using Hamiltonian paths

But… the reads are faulty!
●Learn algorithms to prune errors

Towards neural genome assembly
Pre-train on synthetic graphs…
...generalises to real organisms!
(Still preliminary, but encouraging!)

Further insight: Algorithmic reasoning
If you would like to know more details about teaching GNNs to be more “algorithmic”:

https://www.youtube.com/watch?v=IPQ6CPoluokhttps://drive.google.com/file/d/1_EQ9Yu7VEkvr
HaVHl_WbT5ABvxrSNY-s/view?usp=sharing

Broader context: Combinatorial Optimisation
Our 43-page survey on GNNs for CO!
https://arxiv.org/abs/2102.09544
Section 3.3. details algorithmic reasoning,with comprehensive references.

Predicting Patient Outcomes withGraph Representation Learning
Emma Rocheteau*, Catherine Tong*, Petar Veličković, Nicholas Lane and Pietro LiòElectronic Health Records (EHRs) in the ICU●EHRs can provide plentiful information about a patient’s progression○ But not all data contained in there are easy to leverage by deep learning systems!

●Today, we focus on diagnoses.

Diagnosis information is hard to use●Large number of possibilities makes distinguishing patterns of comorbidity difﬁcult.

●There is a lack of data for rarer combinations.
○ A long tail of rare diagnoses, difﬁcult for deep learning models to leverage!

Distribution of diagnoses in eICU

The “pattern recognition” method●

Commonly, the “long tail” of diagnoses is discarded and the rest embedded.
○ But this long tail often holds the most useful cues, which diagnosticians regularly use!

●

How do clinicians often make decisions about diagnoses or prognoses?

●

The pattern recognition diagnostic method, as described by Wikipedia:“In a pattern recognition method the provider uses experience to recognize a pattern of clinicalcharacteristics… This may be the primary method used in cases where diseases are "obvious", orthe provider's experience may enable him or her to recognize the condition quickly.”●

We interpret experience as exploitation of related cases the clinician treated in the past.
○ Hence, the cases form a graph!

These links definitely exist :)The graph of patients●Key assumption: patients with related diagnoses will likely have related prognoses!

●If we use this signal wisely, it can be a great way to regularise our model and makeadvantage of sparse diagnosis data.

How to build the graph?
●

The “relatedness” score between two patients i and j is given by:where:○○○○●

D is a diagnosis matrix (s.t. Di𝜇 means “does patient i have diagnosis 𝜇”?)d𝜇 is the frequency of diagnosis 𝜇m is the number of diagnoses
𝛼 and 𝛾 are hyperparameters

Can threshold based on the relatedness scoresHybrid LSTM-GNN modelOur results

Qualitative: LSTM-GAT Attention weights

AAAI’21 Workshop Recognition

In conclusion...
●Studying biological problems with graph representation learning is likely here to stay
○ Abundance of data “sitting and waiting to be processed”○ In many problems of interest, state-of-the-art is still a shallow method○ Often, biological problems can give rise to core methodological progress.

●With the right mindset, no proper biological training is needed!
○ Just the ability to carefully listen, and work together with biologists.

●For biologists: I hope I’ve convinced you that GNNs could be a useful tool!

●But ultimately, I would love to stimulate, and see even more of, interdisciplinary research.

Thank you!
Questions?
petarv@google.com | https://petar-v.com
With many thanks to:
Lead authors: Guillem Cucurull, Andreea Deac, Stefan Spalević, Lovro Vrček, Emma Rocheteau, Catherine Tong
Bio/chem: Konrad Wagstyl, Estrid Jakobsen, Alan Evans, Pietro Sormanni, Pietro Lio, Mile Šikić, Jovana Kovačević
ML collaborators: Arantxa Casanova, Yu-Hsiang Huang
ML advisors: Adriana Romero, Michal Drozdzal, Yoshua Bengio, Jian Tang,, Mladen Nikolić, Nic Lane