Cross-modal Graph Contrastive Learning with Cellular Images
Authors: Shuangjia Zheng
Jiahua Rao
Jixian Zhang
Cohen Ethan
Chengtao Li
Yuedong Yang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Celluar Image
Drug discovery
Graph neural networks
Self-supervised learning
Cross-modal learning

Abstract: Constructing discriminative representations of molecules lies at the core of a number of domains such as drug discovery, material science, and chemistry. State-of-the-art methods employ graph neural networks (GNNs) and self-supervised learning (SSL) to learn the structural representations from unlabeled data, which can then be fine-tuned for downstream tasks. Albeit powerful, these methods that are pre-trained solely on molecular structures cannot generalize well to the tasks involved in intricate biological processes. To cope with this challenge, we propose using high-content cell microscopy images to assist in learning molecular representation. The fundamental rationale of our method is to leverage the correspondence between molecular topological structures and the caused perturbations at the phenotypic level. By including cross-modal pre-training with different types of contrastive loss functions in a unified framework, our model can efficiently learn generic and informative representations from cellular images, which are complementary to molecular structures. Empirical experiments demonstrated that the model transfers non-trivially to a variety of downstream tasks and is often competitive with the existing SSL baselines, e.g., a 15.4\% absolute Hit@10 gains in graph-image retrieval task and a 4.0\% absolute AUC improvements in clinical outcome predictions. Further zero-shot case studies show the potential of the approach to be applied to real-world drug discovery. 

Under review as a conference paper at ICLR 2023
CROSS-MODAL GRAPH CONTRASTIVE LEARNING WITH CELLULAR IMAGES
Anonymous authors Paper under double-blind review
ABSTRACT
Constructing discriminative representations of molecules lies the core of number of domains such as drug discovery, material science, and chemistry. State- of-the-art methods employ graph neural networks (GNNs) and sulf-supe learning (SSL) to learn the structural representations from unlabeled data, which can then be fine-tuned for downstream tasks. Albeit powerful, these methods that are pre-trained solely on molecular structures cannot generalize well to the tasks involved in intricate biological processes. To cope with this challenge, we propose using high-content cell microscopy images to assist in learning molecular representation. The fundamental rationale of our method is to leverage the corre- spondence between molecular topological structures and the caused perturbations at the phenotypic level. By including tross-modar pre-training with different types of contrastive loss funetions in unified framework, model efficiently learn generic and informative representations from cellular images, which are complementary to molecular structures. Empirical experiments demonstrated that the model transfers non-trivially to variety of downstream tasks and is often competitive with the existing SSL baselines, e.g., 15.4% absolute Hit@ 10 gains in graph-image retrieval task and 4.0% absolute AUC improvements in clinical outcome predictions. Further zero-shot case studies show the potential of the approach to be applied to real-world drug discovery.
INTRODUCTION
Learning discriminative representations of molecules is a fundamental task for numerous applica- tions such as molecular property prediction, de novo drug design and material discovery [Wu et al. (2018). Since molecular structures are essentially topological graphs with atoms and covalent bonds, graph representation learning can be naturally introduced to capture the representation of molccules Duvenaud et al. (2015); Xu et al. (2019); Song et al. (020)): Ying et al. (202I). Despite the fruitful progress, graph neural networks (GNNs) are known to be data- auta-hungry.oo i.c., requiring a large amount of labeled data for training. However, task-specific labeled data are far from sufficient, as wet-lab experiments are resource-ingg inource-intensive and time-consuming As result, training datasets in chemistry and drug discovery are typically limited in size, and neural networks tend to overfit them, leading to poor generalization capability of the learned representations.
Inspired by the fruitful achievements of the self-supervision learning in computer vision Chen et al. (2020a); He et al. (2020) and natural language processing Devlin et al. (2018) Radford et al. (2018). there have been some attempts to extend self-supervised schemes to molecular representation learning and lead to remarkable results Hu et al. (2020b); Xu et al. (202)); You et al. (020 202T); Rong et al. (2020): Velickovic et al. (019). The core of self-supervised learning lies in establishing meaningful pre-training objectives to harness the power of large unlabeled data. The pre-trained neural networks
can then be used to fine-tune for scall-scale downstream tasks.
However, pre-training on molecular graph structures remains a stiff challenge. One of the limitations of current approaches is the lack of domain knowledge in chemistry or chemical synthesis. Recent studies have pointed out that pre- trainee GNNs with random node/edge masking gives limited im- provements and often lead to negative transfer downstream tasks Hu al. (2020b); Stärk et al. (2021), as the perturbations actions on graph structures can hurt the structural inductive bias of molecules. Furthermore, molecular modeling tasks often require predicting the binding/interaction

---

Under review as a conference paper at ICLR 2023
(a) High Content Screening
(b) Samples of Mol-Image Pairs
Molecular Pertubations
Cell Morphological Changes
Molecules
High-content Cell Images
Figure I: (a) Illustration of high-content screening experimental protocol. (b) Samples of molecule- images pairs. Note that one molecule produces multiple images.
between molecules and other biological entities (e.g., RNA, proteins, pathways), and further gen- eralizing to the phenotypic/clinical outcomes caused by these specific bindings Kuhn et al. (016). Self- self-supere learning methods that solely manipulate molecular structures struggle handle downstream tasks that involve complex biological processes, limiting their practicality in wide range of drug discovery applications.
To this end, we propose using high-content cell microscopy images to assist learning molecular representation, extending the molecular representation beyond chemical structures and thus, improv- ing the generalization capability. High-content cell microscopy imaging (HCT) an increasingly important biotechnology recent years in the domain of drug discovery and system biology Chan drasekaran et al. (202T); |Bray et al. (2016). As shown in Figure | small molecules enter into cells and affect their biological functions and pathways, leading morphological changes in cell shape, number, structure, etc., that are visible in microscopy images after staining. Analysis and modeling based these high-content images have shown great success in molecular bioactivity prediction Hofmarcher et al. (2019). mechanism identification Caicedo et al. (2022), polypharnacology predic- tion Chow al. 2022). 2. etc. Thus. we hypothesize that this phenotypic modality has complementary strengths with molecular structures to make enhanced representations and thus benefit the downstream tasks involved in intricate biological processes.
However, building connections between molecular structures and high-content cellular images is challenging task that highlights representative, unsolved problems in cross cross-modal learning. The first challenge comes from the unclear fine-grained correspondence between molecules and images. Unlike conventional cross-modal paired data such caption and picture, the patterns of the molecule are not directly reflected in the cellular image, thus preventing us from using traditional cross- modal encoders for alignment. The second challenge arises from the noise and batch effect of the cellular data. For example, cellular images obtained from the same molecule can vary considerably. Existing cross-mode modal pre-trate pre-training objectives may overfit the noisy images and decrease the model's generalization ability.
Herein, we propose Molecular graph and high content imaGe Alignment (MIGA), a novel cross- modal aeth--ndd-mmage pre-training framework to address the above issues. We first encode the molecule and cell microscopy images independently with molecular graph encoder and an image encoder. Then align the graph embeddings with the image embeddings through three contrastive modules: graph-image graph-image matching (GGIM). Specifically, (i) GIC encourages high similarity between the latent embeddings of matched
image contrastive (GIC) learning, masked graph modeling (MGM) and generative
pairs while pushing those of non- non-matched pairs apart; (ii) MGM, local cross-modal module that utilizes both the observed (unmasked) graph and the cell image to predict the masked molecular patterns and (iii) GGIM aims to further distinguish the hard negative graph-image pairs that share similar global semantics but differ in fine- ame-grained details. The three modules complementary and thus the combination of these modules the encoders to perform cross-modal learning by capturing structural and localized information; (2) learn common oowdimensional space embed graphs and images that are biologically meaningful. Enabled by the massive publicly available high-content screening data Bray et al. (2017), we establish novel cross-modal benchmark dataset that contains 750k molecular graph-cellular image pairs. To evaluate models on this benchmark, we propose new biological meaningful retrieval task specific to
1) make it easier for
2

---

Under review as a conference paper at ICLR 2023
graph-image cross-modal learning. We also include existing clinical outcome prediction and property prediction tasks to further assess the learned representations. Extensive experiments demonstrate that the cross-mns modal representations learned by our proposed model, MIGA, benefit wide range of downstream tasks that require extensive biological priors. For example, MIGA achieves a 15.4% absolute Hit@@ 10 gain in graph-image retrieval task and 4.0% absolute AUC improvement in clinical outcome predictions, respectively, over existing state-of-the-an methods.
2 RELATED WORK
Self-supervised learning on graphs. Graph self-supervised pre-learning attempts to obtain su- pervision in unlabelled data to learn meaningful representations that be further transferred to downstream tasks You et al. (2020); Hu et al. (2020b c); Xu et al. (202 12 You et al. (2021) Rong al. (2020); Velickovic et al. (2019); [un al. (2019); Zhang et al. (2020); Liu et al. (2021a);
Stärk et al. (2021). In general. these methods fall into two categories: contrastive- based methods and senerative-basee methods Liu al. (2021b). The former aims to generate different views from the original graph and perform intra-graph contrastive learning Hu et al. (2020b): Xuu et al. (2021); You al. (2200) 2021> ); Velickovic et al. (2019), while the latter ones are trained in a supervised manner to generate masked sub- bub--paterns or attributes at the inter nnter-graph level Hu et al. (2020b); [Rong] ict al. (2020); Zhang et al. (2020). These approaches achieve remarkable performance on molecular graph representation tasks, but lack the ability to predict the complex properties involved in intricate biological processes.
Cross-modal pre-training. Pre-training strategies for multi-modal tasks have attracted massive attention, with most of these efforts targeting Visual-Language representation learning. Most of them can be grouped into two categories. The first category to use multi-modal encoders capture the interaction between image and text embeddings Su et al. (2019); Lu et al. 2020);; Chen et al. (2020b): Li et al. (2020); Huang et al. (202T); Zhang et al. (2021). Approaches in this category achieve remarkable performance, but most of them require righ-quality images and pre- pre-trained object detectors. The other category focuses on learning independent decoders for different modalities Radford al.
[2021) Jia et al. (2021); Xu al. (002);; Taleb et al. (2022). For instance, CL.IP Radford et al.
2021 learns pairwise relationships between language and images by performing pre-training on large amount of web data using independent encoders connected by contrastive losses. More recently, CLOOME (Snchez- Ferneenezeranzzzzz et al. (2022) employs CLIP-like loss to pre- train the cellular image encoder with molecular fingerprints. Different from their focus the image encoder, MIGA aims at extracting information from cellular images to make enhanced molecular representations and designed two complementary cross- modal contrastive losses for information interaction in an end-to-end manner.
3 METHODS
At the core of our method is the idea of infusing structural representations with biological perception by building the connections between molecular graph and their induced morphological features.
To achieve this, as illustrated in Figure given pairs graph and image, we employ two independent encoders (a GNN f and a CNN f*) to produce the representations of a molecular graph G and cellular image 7, and align them with inter- and intra- contrastive losses. This cross-modal framework pulls the matched graph-image pairs together and contrasts the unmatched pairs apart. After pre- training, we use the output representations to make zero-shot predictions or fine-tune the networks on downstream tasks. This cross-modal learning process can also be interpreted as morphological information being passed through the convolutional neural network to the graph neural network in a knowledge distillation manner Tian et al. (2019).
In the following sections, we first introduce the structural and image encoders for cross-modal learning (section 3.I). Then we depict the details contrastive training objectives (section 3.2).
followed by the experimental setting (section 4.1) and results (section 4.2)) 4.2).
3

---

Under review as a conference paper at ICLR 2023
Graph--magg Conrastive
Graph-lmage Retrieval
Graph Encoder
Image Encoder
S
Success
Falure
Graph Modeling
Prediction
Dr
Du
Dio
D2
D2
Day
Physical Chem
Dur
Dr
Du
Duc
hysiology
Biophysics
Dus
Deu
Der
Dan
Generative Graph-lmage Matching
Property
(a) Pre-training
(b) Downstream Tasks
Figure 2: Overview of our method, MIGA. Molecular graph and cellular image representations are jointly learned from pairwise data. We used three complementary contrastive objectives to perform cross-modal pre-training. The learned representations can be used for graph-image or graph-only downstream task transfer.
3.1 STRUCTURE AND IMAGE ENCODERS
compound structure can represented as an attributed graph G -V,, where |V| = denotes a set of n atoms (nodes) and |&| denotes set of m bonds (edges). We represent X, E gnxd.. for the node attributes with d,, as the the feature dimension node and Euv mm for the edge attributes with de the feature dimension of edge. A graph neural network (GNN) f learns to embed an attributed graph into feature vector zg. We adopt the Graph Isomorphism Network (GIN) from Xu et al. (2019), where the node and edge attributes propagated at each iteration. Formally, the k-th iteration of GNN is:
Structural encoder.
(((h) 9t
(X.)) (I:) (n( Pxda ((h(--) (((-"),,,))) u E N())))
where (k) are the representation of node v at the k-th (uyer,N(v) is the neighbourhood set of node (k) v, (0) is initialised with X., encoding its atom properties. D'GG stands for the aggregation function and g stands for the update function. After K graph convolutions, h* have captured their K K-hop neighbourhood information. Finally, readout function is used to aggregate all node representations output by the K-th GNN layer to obtain the entire molecule's representation 2g:
ZG Σ READOUT(H(K)
VEV
Image encoder. For the cellular image 1, we first use residual convolutional neural networks (ResNet-34) He et al. (2016) as the basic feature extractor f because is lightweight, widely-adopted and have proven performance. We also implement DenseNet Huang et al. (2017), EfficientNet Tan & Le (2019) and the recently proposed Vision Transformer (VIT) Dosovitskiy et al. (020) as comparisons. We strictly follow their original implementation with small modification, namely adding an additional projection layer before the final output. These models are initialized with weights pre-trained on ImageNet Deng et al. (2009). Each input image is encoded into one-dimensional feature vector for further fusion.
3.2 PRE-TNAINING FRAMEWORK
We pre-trained MIGA through three contrastive objectives: graph-image contrastive (GIC) learning, masked graph modeling (MGM) and generative graph-image matching (GGIM).
4

---

Under review as a conference paper at ICLR 2023
Graph-image contrastive (GIC) learning aims pull embeddings of the matched molecule- image pairs together while pushing those of unmatched pairs apart by maximizing a lower bound on the mutual information (MI) between the molecular graph and cellular image for the positive pairs. We achieve this by minimizing symmetric InfoNCE. loss Van den Oord et al. (2018) to maximize a lower bound on T). Formally, the graph-image contrastive loss is defined as:
LGIC - BMG.n log (o0(((z,=z)//1 + log exp(s(21,
(3)
Ekzi exp((((1
Here, the similarity function s s(2g. (z)) fw (21)ll2 where f. and fw are two linear projection layers that embed representations to a common space. T a temperature parameter, K is set of negative image samples that not matched to G.
Masked graph modeling (MGM). To simultaneously leverage the intra-molecular graph informa- tion and strengthen the interaction between molecule and image, we further utilize both the image and the partial molecular graph to predict the masked sub-patterns. Following the masking strategies Hu et al. (2020b), we randomly mask the atom/bond attributes and constructed the context graphs for each molecular graph. We then use the surrounding partial graph structures along with the corresponding image information to predict the masked attributed subgraphs and the corresponding attributes. Our goal is to pre-train molecular GNN f9 that can not only learn the context information of atoms in similar structures but also capture domain knowledge by learning the regularities of the node/edge attributes distributed over graph structure. Herein, we defined the masked molecular graph as Gmak and the observed (unmasked) graph as Gobs Therefore, the training graph-ine image pair (G. has been transformed to (Gmek, Gobe I) and the objective of MGM is:
LMGM = E(Gmmsk,CH GGobe ID) Plog ((Gtssk (Gobs, [))]
where (Gmskk Gobbs II) is randomly sampled MGM training is equivalent to optimizing the cross-entropy loss: :
pair from the training set D. Thus, the
LMGM =
1
Σ
loglp(Gmaa (9k x1,)
(5)
D
(Og
where the gruss is the prediction from the observed graph ghds and image embedding 2,. The function fm is the molecule attributes and context graphs prediction model. Our MLM training is minimized by cross-entropy loss because g mak is one-hot vocabulary distribution where the ground-truth attributes/subgraphs has a probability of 1.
Generative Graph-image matching (GGIM) aims further distinguish the hard negative graph- image pairs that share similar global semantics but differ in fine-grained details. This is achieved by the combination graph-image matching (GIM) loss and a generative matching (GM) loss.
Inspired by Li al. 2, we firstly utilize cross-modal encoder Se fuse two unimodal representations and produce joint representation of the graph-image pair and append multilayer perceptron followed by softmax to predict the matching probability of pm fo(aG, This can be
formulated as:
CarM (log)) 1))
(6)
where yd is the ground truth label representing whether the graph and the image are matched or not matched. The expected log- liglliill function could be defined as:
P+N
LArM
1
Σ
Wi Th log(po)) + (1 - =x" log(1 pF
P+N
where P denotes the number of positive pairs and N denotes the number of negative pairs. In practice, P: = 1: 1. We use contrastive similarity (Eq, 3 to sample the in-batch hard negative pairs, where the molecular embeddings that are more similar to the image have a higher chance of being sampled (and vice versa). These pairs are hard to distinguish because there may exist other images different from the ground-truth that reflects the molecular perturbation equally well (or better) because of the batch effect Tran et al. (2020).
Motivated by recent success in generative contrastive learning Parmar et al. (2021) Liu et al. (2021a), further employ variational auto-encoders (VAE) Kingma & Welling (2013) generative agents
5

---

Under review as a conference paper at ICLR 2023
to reduce noise from experiments and enhance the cross-modal interaction among the hard negative pairs. In particular, the generative agents are asked to recover the representation of one modality given the parallel representation from the other modality. Herein, performed cross-modal generation from two directions. This GM loss function can be formulated as:
LGM Aul (DKu(9
(8)
+ [B (==l=== loggoo + (oope//
where AuI is the hyperparameter balancing the reconstruction loss. p(2r) and p(za) is the prior of image embedding and graph embedding, respectively, and (o is the corresponding posterior. Finally, the full contrastive objective of MIGA is:
L LGIC + LMGM + LGIM + LGM
4 EXPERIMENTS
4.1 DATASETS AND TASKS
Pre-training Dataset. We perform our experiments on the Cell Painting dataset CIL introduced by Bray et al. (0166; 2017). The dataset originally consists of 919,265 cellular images collected from 30,616 molecular interventions. Each image contains five color channels that capture the morphology of five cellular compartments: nucleus (DNA), Endoplasmic reticulum (ER), mucleolus/cytoplasmic RNA (RNA), F-actin cytoskeleton (AGP) and Mitochondria (Mito). A molecular intervention is photographed from multiple views in an experimental well and the experiment was repeated several times, resulting in an average of 30 views for each molecule. In order keep the data balanced, we restricted each molecule to a maximum of 30 images and removed the untreated reference images, resulting in a cross-modal graph-image benchmark containing 750K views. We refer this benchmark as CIL-750K. Detailed pre-process procedure and analysis of the pre-train dataset can be found in Appendix A.
Downstream Tasks.
We evaluate the pre-trained model in three downstream tasks: graph-image retrieval, clinical outcome prediction and molecular property prediction.
Graph-Image Retrieval contains two tasks: (1) graph as query and image as targets (Image retrieval); (2) image as query and graph as targets (graph retrieval/scaffold hopping). The goal of this task is to demonstrate whether the learned embeddings are able to preserve the inherent relations among corresponding graph-image pairs. We randomly split the CIL-750K dataset into training set of 27.6K molecules corresponding to 680K images, and hold out the remaining data for testing. The held-out data consists of 3K molecules and the corresponding 50K images. We formulate the retrieval task as a ranking problem. In the inference phase, given a query molecular graph in the held-out set, take images in the held-out set candidate pool and rank candidate images according to the 1.2 distance between the image embeddings and the molecular embeddings, and vice versa. The ranking of the ground truth image/graph can then be used to calculate AUC, MRR (mean reciprocal rank), and Hit@| 1, 5, 10 (hit ratio with cut-off values of 1, 5, and 10). Experiments are performed 5 times with different seeds and the average results are reported.
Clinical Trial Outcome Prediction aims to predict the clinical trial outcome (success or failure) of drug molecules. This task is extremely challenging as requires external knowledge of system biology and trial risk from the clinical record. We use the Trial Outcome Prediction (TOP) benchmark constructed by Fu et al. 2022 for model evaluation. After dropping the combination mediation, the benchmark contains 944, 2865 and 1752 molecules for Phase I, II, and III tasks, respectively. We follow the data splitting proposed by [Fu et al. (2022) and employ Precision- recall area under the curve (PR-AUC) and Area under the receiver operating characteristic curve (ROC-AUC) to measure the performance of all methods.
Molecular Property Prediction. We further evaluate MIGA on six molecular property datasets: HIV, Tox21, BBBP,ToxCast, ESOL and Lipophilicity. These datasets are introduced by Wu et al. (2018) and further benchmarked by OGB communityHHu
al. (2020a) for low-resource graph representation learning. Each data set contains thousands of molecular lar graphs as well as binary/scalar labels indicating the property of interest. We follow the OGB Hu et al. (2020a) setting and adopt the scaffold splitting strategy with a ratio for vuli//vali//test as 8:1:1 during fine-tuning.

---

Under review as a conference paper at ICLR 2023
Task Metrics
Image Retrieval Hit@1
Gruph Retricval Hit@)
MRR
AUC
Hit@5
Hit@ 10
MRR
AUC
Hit@5
Hit@ I0
Random Init
0.051
0.500
0.010
0.050
0.100
0.051
0.500
0.010
0.050
0.
BCFP+CelIProfiler
0.052
0.511
0.010
0.049
0.100
0.053
0.500
0.011
0.050
0.100
GraphCL
0.247
0.818
0.106
0,391
0.558
0.279
0.835
0.132
0.433
0.616
CLIP
0.265
0.890
0.105
0.437
0.635
0.327
0.878
0.171
0.515
0.672
CLOOB
0.278
0.895
0.137
0.451
0.612
ALIGN
0.288
0.821
0.434
0.379
0.810
0.214
0.581
0.713
GIC
0.288
0.847
0.148
0.434
0.594
0,280
0.907
0.125
0.447
0.652
GIC+GIM
0,303
0.876
0.145
0.485
0.660
0,337
0.927
0.176
0,523
0.693
GIC+MGM
0.409
0.913
0.244
0.612
0.741
0.405
0.931
0.244
0.598
0.737
GIC+MGM+GIM
0.401
0.926
0.230
0.616
0.743
0.433
0.935
0.275
0.628
0.739
MIGA (Pull)
0.417
0.936
0.248
0.623
0.748
0.413
0.940
0.249
0.614
0.739
Table I: Graph-image retrieval tasks on held-out set of CIL-750K. MIGA and its variants are compared with adapted pre-training methods, GraphCL, You al. (2020), CLIP|Radiord et al. (202T),
CLOOB
et al. (2022), ALIGN Jia et al. (2021 along with random initiali initialization
and feature-based method ECFP ECF-PRogees & Hahn (20T0)
CellProfiler Carpenter et al. (2066).
*means re-implemented. The average of MRR, AUC, Hit@ 1. H and Hit 10 are reported. The best and second best results are marked bold and bold, respectively.
Image Retrieval
Graph Retrieval
Active mot Inective mol
X
TP53
ALIGN
X
X
Ground-trulti truth
Quinn
Figure 3: Case study on image retrieval task (left) and zero-shot image retrieval task (right), The images retrieved by our method and baseline method are shown on the left. Right side shows the cells induced by cDNA our model can find diverse molecules that have similar functions to cDNA interventions (ticked). The full version is in Appendix D.
4.2 RESULTS
4.2.1
GRAPH-IMAGE RETRIEVAL
Baselines. Since we present improvements to pre-training models, we primarily compare our approach the existing mfate-of-the-ee graph SSL method, You et al. (2020) and cross- modal pre-learning methods, CLIP Radford et al. (202T), (CLLOO/Sanchez-FFF SSanchez-Ferandee and ALIGN Jia et al. (00TI). As none of them were specifically designed for graph-image cross modal learning, we adapted their encoders but followed their loss functions and tricks to perform pre-training on CIL-750K. We then evaluate the effectiveness of the proposed pre-traiming objectives (i.e., GIC, MGM, GIM, GGIM) with four variants of our method. For each baseline, trained model to output embeddings of molecular graphs and cellular images, then rank the candidate pool based their similarity. We also include random initialization and feature angineering-baseee representation (ECFP [ECFP|Rogers & Hahn (2010) + CellProfiler Carpenter et al. (2006)) as baselines demonstrate the challenge of the task. More details and hyperparameters have been attached in Appendix B.
et al. (2022)
the pre-
Result on CIL-750K held-out set. Results of graph- retrieval tasks are shown in Table I is clear that method MIGA significantly outperforms baseline methods including ECFP+CellProfiler, GraphCL, CLIP and ALIGN. For example, MIGA achieves 12.9% absolute MRR gain and 15.4% absolute Hir@ 10 gain the best baseline ALIGN the image retrieval task. These improvements
7

---

Under review as a conference paper at ICLR 2023
Task
Phase
Phase
Phase
Metries
PR-AUC
AUC
PR-AUC
ACC
PR-AUC
AUC
LR
0.634 (0.007)
0.487 (0.006)
0.509 (0.014)
0.534 (0.017)
0.675 (0.010)
0.528 (0.003)
RF
0.651 (0.013)
0.488 (0.009)
0.488 (0.005)
0.523 (0.009)
0.722 (0.011)
0.588 (0.013)
XGBoost
0.646 (0.003)
0.508 (0.006)
0.481 (0.004)
0.516 (0.007)
0.712 (0.009)
0.597 (0.015)
HINT
0,683 (0.015)
0.516 (0.005)
0.537 (0.004)
0.584 (0.003)
0.689 (0.003)
0.621 (0.006)
ContextPred
0.693 (0.006)
0.541 (0.019)
0.544 (0.019)
0.586 (0.003)
0,710 (0.023)
0.554 (0.036)
GraphLoGi
0,681 (0.016)
0.539 (0.016)
0,550 (0.043)
0.593 (0.043)
0.719 (0.024)
0.554 (0.024)
GROVER
0.711 (0.015)
0.559 (0.024)
0.521 (0.005)
0.574 (0.011)
0.713 (0.013)
0.575 (0.028)
GraphCI.
0.721 (0.020)
0.578 (0.018)
0.543 (0.008)
0.588 (0.004)
0.733 (0.011)
0.601 (0.008)
JOAO
0.736 (0.019)
0.586 (0.018)
0.546 (0.018)
0.587 (0.000)
0.720 (0.000)
0.563 (0.006)
MIGA
0.758 (0.010)
0.601 (0.031)
0.562 (0.010)
0.605 (0.022)
0.729 (0.008)
0.654 (0.016)
Table 2: Performance comparison of MIGA and several baselines for phase-level-outcomee predictions on TOP dataset. We report the mean (and standard deviation) PR-AUC and ROC-AUC of five times. The best and second best results are marked bold and bold, respectively.
are consistent in the graph retrieval tasks, proving that the proposed method has superior performance in graph-image cross-modal learning. Compared to the basic loss (GIC), adding MGM and GIM both substantially improves the pre-trained model's performance across two tasks. This is in line with our expectation as GIC operates on unimodal representations while MGM and GIM operate on cross-modal representations. The proposed GGIM further enhances the model by reducing the noise and building explicit connections between two modalities.
Case study for image retrieval. We randomly select 20 pairs in the held-out set for case study, Figure 3hows the results of four molecules, while the full version is attached in Appendix C. The result demonstrates that our model is more powerful in prioritizing the images that are similar to the ground- -ouun compared to ALIGN.
Case study for zero-shot graph retrieval. Of more practical value in the real world is the use of cellular images induced by non-small molecule interventions (e.g. cNDA Rohban et al. (2017)) 2011)
to recall small molecules with similar effects. We collect 6 sets of cellular image induced by cDNA interventions for specific genes from Rohban et al. (2017). We use EXCAPEDB (Sun et al. (017)) to retrieve active (agonists) and inactive molecules these six genes as the candidate pools. Each gene has 20 active and 100 inactive molecules. Specifically, the mean Hit@ 10 for MIGA was 0.49 = 0.05 much higher than the random enrichment 0.167 (20/120), demonstrating the ability of our method to generalize to new datasets in zero-shot manner. This result also indicates the practical potential of our method in virtual screening and drug repurposing. Figure3]show. the results of two cellular images with cDNA-induced gene over-expression, the top-6 molecules ranked by MIGA are shown, see Appendix D for the full version.
4.2.2 CLINICAL TRIAL OUTCOME PREDICTION
Baselines. Since clinical prediction is a single-modality task, we compare the proposed method with the state-of-the-art SSL methods, ContextPred Hu et al. (2020b), GraphLoG Xu et al. (2021), GROVER Rong et al. (2020). GraphCL You et al. (2020) and JOAO You et al. (2021). D. We also
included three machine learning-basse based methods (RF, LR, XGBoost) and a knowledge-awee aware GNN model HINT as listed in Fu et al. (2022).
Results. Results are summarized in Table 2 Our best model MIGA improves PR-AUC by an average of 1.67% and AUC by an average of 3.97% on the three clinical outcome prediction datasets, respectively, compared with the best SSL baseline JOAO. This is consistent with our hypothesis that phenotypic features can help predict tasks that involve complex biological processes.
4.2.3 MOLECULAR PROPERTY PREDICTION
Baselines. We compare the proposed method with the SOTA SSL methods including EdgePredHamilto et al. (2017). ContextPred Hu et al. (20205). Attraskknnguu et al. (20206). GraphCI, You et al. 2020). InfoCrapph Sun et al. (019)) GROVERRong et al. (0020), Graphl
8

---

Under review as a conference paper at ICLR 2023
Classification (AUC) ToxxCst
Regression (RMSE)
Dataset
HIV
Tox21
BBBP
Avg-
ESOI.
Lipo
Avg.
Non-pretrain
70.30 (0.51)
68.90 (0.80)
58.60 (1.20)
65.40(2.4)
65.80
1.278 (0.24)
0.744 (0.14)
1.011
ContextPred
74.17 (1.33)
71.44 (0.11)
60.05 (0.15)
69.87 (0.99)
68.88
1.141 (0.03)
0.724 (0.02)
0.933
AuteMusk
75.55 (1.00)
74.58 (0.66)
59.51 (0.36)
68.88 (2.65)
69.63
1.194 (0.04)
0.736 (0.02)
0.965
EdgePred
72.53 (1.20)
68.86 (0.38)
$7.39 (0.80)
63.45 (1.39)
65.56
(0.05)
0.751 (0.02)
0,949
InfoCiraph
76.22 (0.24)
69.22 (0.78)
59.87 (0.35)
63,75 (1.52)
67.27
1.242 (0.01)
0.725 (0.01)
0.984
GraphLoG
73.29 (2.64)
69.80 (0.41)
59.22 (1.05
68.43 (2.86)
67.68
1.194 (0.02)
0.766 (0.01)
0.980
GraphCI.
74.85 (1.71)
74.19 (0.43)
61.37 (0.10)
66.13 (1.68)
69.14
1.151 (0.04)
0.745 (0.02)
0,948
GROVER
74.35 (0.92)
74.02 (0.79)
61.30 (0.13)
69.88 (0.58)
69.89
1.199 (0.02)
0.721 (0.01)
0.960
JOAO
74.91 (0.66)
74.60 (0.49)
61.62 (0.37)
68.33 (0.58)
69.87
1.117 (0.05)
0.753 (0.02)
0.935
MIGA
76.38 (0.55)
75.23 (0.71)
62.34 (0.23)
71.52 (0.43)
71.37
1.123 (0.01)
0.717 (0.00)
0.919
Table 3: Comparison of SSL baselines against MIGA on six OGB datasets. Mean ROC-AUC and
Root Mean Squared Error (RMSE) (with the SD) of 5 times independent test are reported. The best and second best results are marked bold and bold, respectively.
View
MRR
AUC
Hit@1 1
Hit@ 10
Methods
MRR
AUC
Hit@1
Hit@ 10
I
0.336
0.927
0.175
0.692
ResNet
0.413
0.940
0.249
0.739
5
0.419
0.933
0.251
0.728
EffciNet
0.405
0.946
0.236
0.744
10
0.427
0.938
0.263
0.742
DenseNet
0.406
0.941
0.235
0.738
Full
0.413
0.940
0.249
0.739
ViT
0.310
0.919
0.146
0.676
Table 4: Effect of # view per molecule.
Table 5: Effect of CNN architecture.
et al. (2021), GraphCL You et al. (2020), JOAC et al. (2021). All the SSL models are pre-trained using released source code on CIL-750K data set. For line-tuning, we follow the same setting with You et al. (2020 2021)
Results. Results are summarized in Table 3 As we can see, MIGA performs the best on 5 out of 6 datasets and outperforms the existing SSL methods the average performance by large margin. The slightly lower performance in ESOL is expected as this property is related to the molecule's 's intrinsic characteristics, e.g., hydrophilicity and hydrophobicity. It does not rely on biological processes and phenotypic changes. Even though, MIGA will not cause negative transfer compared non-pretrain baseline.
4.3 ABLATION STUDY
Table 4 shows the effect the number of view per molecule perform on graph retrieval task. We notice that with less than 10 views, the more views involve pre-training the better the results would be, but using all views (on average 25 views per molecule) does not achieve more gains. We attribute this phenomenon to the batch effect of the cellular images, where images from different wells can vary considerably because of the experimental errors and thus mislead the model.
Table 5 studies the impact of CNN architecture choices on the graph retrieval task. Due to the relatively small amount of data, use small CNN variants (ResNet3<(He al. (2016), EfficientNet-
B Tan & Le Le (201),, DenseNet1 2 [Huang et al. 2017) and ViT Disoviiiiiiissiiy et al. (2020) for evaluation. We note that small CNN models such as ResNet, EfficientNet and DenseNet achieve comparable performance while bigger models like ViT do not show superiority. We assume this is because these models are pre-trained on non-cellular images, so heavier models not necessarily lead to gains on cellular tasks.
CONCLUSIONS
In this paper, we propose a novel cross-modal graph pre-training framework, called MIGA. By intro- ducing phenotypic features of cellur images during pre-training, MIGA is able to bring generalisable biological priors to graph neural networks, enabling them to be transferred to downstream biologically relevant tasks. Future directions include designing specific encoders for cellular images, optimizing cross-modal fusion mechanisms and extending the seale the dataset to improve the performance of graph pre-training.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mark-Anthony Bray, Shantanu Singh, Han Han, Chadwick Davis, Blake Borgeson, Cathy Hartland. Maria Kost-Alimova, Sigrun M Gustafsdottir, Christopher C Gibson, and Anne E Carpenter. Cell painting, a high-content amage-baser assay for morphological profiling using multiplexed fluorescent dyes. Nature protocols, 1199:1757-1774, 2016.
Mark-Anthony Bray, Sigrun M Gustafsdottir, Mohammad H Rohban, Shantanu Singh, Vebjomn Ljosa, Katherine L Sokolnicki, Joshua A Bittker, Nicole Bodycombe, Vlado Dančík, Thomas Hasaka, al. A dataset of images and morphological profiles of 30 000 small-molecule treatments using the cell painting assay. Gigascience, 6(12):giw014, 2017.
Juan C Caicedo, John Arevalo, Federica Piccioni, Mark-Anthony Bray, Cathy L Hartland. Xiaoyun Wu, Angela N Brooks, Alice II Berger, Jesse S Boehm, Anne E Carpenter, et al. Cell painting predicts impact of lung cancer variants, Molecular Biology of the Cell. pp. mbc E21, 2022.
Anne Carpenter, Thouis R Jones, Michael R Lamprecht, Colin Clarke, In Han Kang, Ola Friman, David Guertin, Joo Han Chang, Robert A Lindquist, Jason Moffat, et al. Cellprofiler: image analysis software for identifying and quantifying cell phenotypes. Genome biology, 1010):1-11,200
Srinivas Niranj Chandrasckaran, Hugo Ceulemans, Justin D Boyd, and Anne B Carpenter. masge-based profiling for drug discovery: due for lechine-leariin upgrade? Nature Reviews Drug Discovery, 20(2):145-159, 2021.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597- 1607-1607. PMLR, 2020a.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan. Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pp. 104-120. Springer, 2020b.
Yuen Ler Chow, Shantanu Singh, Anne E Carpenter, and Gregory P Way. Predicting drug polypharmacology from cell morphology readouts using variational autoencoder latent space arithmetic. PLoS computational biology, 18(2);e 1009888, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenct: A large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXi: 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom. Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1099, 2020.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru- Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. In Advonces neural information processing systems. pp. 2224 2232, 2015.
Tianfan Fu, Kexin Huang, Cao Xiao, Lucas M Glass, and Jimeng Sun. Hint: Hicrarchical interaction network for orttooo predictions. Patterns, 3(4):100445, 2022.
William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive Representation Learning on Large Graphs, arXiv Jun 2017.
e-prints, art.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778. 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Markus Hofmarcher, Elisabeth Rumetshofer, Djork-Ame Clevert, Sepp Hochreiter, and Gunter Klambauer. Accurate prediction of biological assays with high-throughput microscopy images and convolutional networks. Journal of chemical information and modeling, 19(3):1163-1177 2019.
10

---

Under review as a conference paper at ICLR 2023
Weihua Hu, Matthias Fey. Marinka Zitnik, Yuxiao Dong. Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118 22133,, 2020a.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay S. Pande, and Jure Leskovec. Strate- gies for pre-training graph neural networks. In 8th International Conference on Learning Representations. ICLR. OpenReview.net, 2020b.
Ziniu Hu, Yuxiao Dong, Kuansan Wang. Kai-We Kei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1857 1867, 2020c.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinherger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700 4708, 2017.
Zhicheng Huang, Zhaoyang Zeng. Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-to-end pre-training for vision-language representation learning. Proceedings of the IEEE/CVF Conference on Compuder Vision and Pottern Recognition, pp. 12976-12985, 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904 4916. PMLR, 2021.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 131.:111/2.1114 2013.
Michael Kuhn, Ivica Letunic, Lars Juhl Jensen, and Peer Bork. The sider database drugs and side effects. Nucleic acids research, 44(D1): DD079, 2016.
Gen Li, Nan Duan, Yuejian Fang. Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training, In Proceedings of the AAAI Conference on Artificial Intelligence. volume 34, pp. 11336-11344 2020.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems, 34, 2021.
Shengchao Liu, Hanchen Wang. Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre-training molecular graph representation with 3d geometry. arXiv preprint arXiv:2110.07728. 2021a.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang. Jing Zhang, and Jie Tang. Self-supervised learning: Generative contrastive. IEEE Transactions on Knowledge and Data Engineering, 2021b.
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. In Proceedings of the IEEF//VF Conference on Computer Vision and Pattern Recognition, pp. 10437-10446, 2020.
Gaurav Parmar, Dacheng Li, Kwonjoon Lee, and Zhuowen Tu. Dual contradistinctive generative autocncoder. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 823 832, 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 8748-8763, PMLR, 2021.
David Rogers and Mathew Hahn. Extendedd-oomeecivitty fingerprints. Journal of chemical information and modeling, 50(5):742-754, 2010.
Mohammad Hossein Rohban, Shantanu Singh, Xiaoyun Wu, Julia B Berthet, Mark-Anthony Bray, Yashaswi Shrestha, Xaralabos Varclas, Jesse S Bochm, and Anne Carpenter. Systematic morphological profiling of human gene and allele function via cell painting. Elife, 6:e24060, 2017.
Yu Rong, Yatao Bian, Tingyang Xu. Weiyang Xie, Ying Wei, Wenbing Huang. and Junzhou Huang. Self- supervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems, 13512559-12571, 2020.
11

---

Under review as a conference paper at ICLR 2023
Ana Sanchez-Fernandez, Elisabeth Rumetshofer, Sepp Hochreiter, and Günter Klambauer. Contrastive learning of image-and structure- based representations drug discovery. ICLR2022 Machine Learning for Drug Discovery, 2022.
Ying Song.
Shuangjia Zheng, Zhangming Niu, Zhang- Hua Fu, Yutong Lu, and Yuedong Yang. Communicative representation learning on attributed molecular graphs. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, (IJCAI 2020), pp.
2020.
Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pictro Liò. 3d infomax improves gnns for molecular property prediction. arXiv preprint 10.04221 2021.
Weijie Su, Xizhou Zhu, Yue Cao. Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. V1-bert: Pre-training of generic Visual-linguistic representations. arXiv preprint arXiv:/908.088 2019.
Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- sumi-supe graph-level representation learning via mutual information maximization. arXiv preprint arXiv: 2019.
Jiangming Sun, Nina Jeliazkova, Vladimir Chupakhin, Jose-Felipe Golib-Dzib, Ola Engkvist, Lars Carlsson, Jörg Wegner, Hugo Ceulemans, Ivan Georgiev, Vedrin Jeliazkov, al. Excape-db: integrated large scale datasct facilitating big data analysis in chemogenomics. Journal of cheminformaties, 9(1):1 2017.
Aiham Taleb, Matthias Kirchler, Remo Monti, and Christoph Lippert. Contig: Self-supervised multimodal contrastive learning for medical imaging with geneties. In Proceedings of the IEEE/CVF Conference on 2022.
Computer Vision and Pattern Recognition, pp.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference machine learning, pp. 6105-6114, 14. PMLR, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint
arXiv: 1910,09990 2019.
Hoa Thi Nhu Tran, Kok Siong Ang, Marion Chevrier, Xiaomeng Zhang, Nicole Yee Shin Lee, Michelle Goh, and Jinmiao Chen. A benchmark of batch-effect correction methods for single-cell ra sequencing data. Genome biology, 21(1):1 32, 2020.
Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pp. 1-Xiv-18077 2018.
Petar Velickovie, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. ICLR (Poster), 2(3):4, 2019.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu. Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learning. Chemical science, 9 (2):513-530, 2018.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18134 18144, 2022.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In 7th International Conference on Learning Representations, ICLR. OpenReview.net, 2019.
Minghao Xu, Hang Wang, Bingbing Ni. Hongyu Guo, and Jian Tang. Self-supervised graph-level representation learning with local and global structure. In Marina Mcila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML, volume 139 of Proceedings of Machine Learning Research, pp. 11548- 11558. PMLR, 2021.
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Kc, Di He, Yanming Shen, and Tic- Yan Liu. Do transformers really perform bad for graph representation? arXiv preprint arXiv: 20022000.00. 05234, 2021.
Yuning You, Tianlong Chen, Yongduo Sui. Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In Hugo I Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tice Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS, 2020.
Yuning You. Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML, volume 1 139 of Proceedings of Machine Learning Research, pp. 12121-12132. PMLR, 2021.
12

---

Under review as a conference paper at ICLR 2023
Pengchuan Zhang. Xiujun Li, Xiaowei Hu, Jianwei Yang. Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. VinvI: Making visual representations matter in vision-language models, arXiv e-prints, pp. arXiv-2101, 2021.
Shichang Zhang, Ziniu Hu. Arjun Subramonian, and Yizhou Sun. Motif-driven contrastive learning of graph representations. aiviv preprint arXiv:20/2./253, 2020.
13


Reviews:
[{"review_id": "h-CQRXJIxBz", "reviewer": "Reviewer_pEPS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work utilizes the self-supervised learning paradigm to improve the data representations in Graph Neural Networks. Different from previous literature, the proposed method employs cross-modal correspondences between the Graphs of molecular structures and the associated (paired) cell microscopy images. The intuition behind combining these specific modalities is clear, drug molecules have the ability to cause perturbations (subtle morphological changes) to cells, e.g. their shapes, numbers, structures, and so on. To that end, the paper proposes a pretraining method called MIGA that processes these paired modalities, and aligns them in the feature space. MIGA is trained with three combined loss terms, which are inspired from existing literature on contrastive learning and generative modeling. Then, the learned data representations, stored in the form of neural network weights in the Graph and Image encoders, are evaluated on several downstream tasks. The chosen downstream tasks seem relevant to the domain, and are sensible overall. The reported results on these downstream tasks confirm the improvements obtained by pretraining by their method, MIGA, as opposed to the baselines from literature.", "summary_of_the_review": "Overall, I recommend accepting the paper. The method is novel, the writing is clear, and the experimental results are sound.", "strength_and_weaknesses": "1- The proposed method, MIGA, utilizes the cross-modal information that exist naturally in microscopy images to improve graph data representations. While the underlying algorithms and losses to perform this cross-modal alignment are not novel, the idea of employing these heterogeneous modalities is inspired and novel. \n2- The paper is easy to understand, and the used language is clear. \n3- The experimental results are comprehensive, and confirm the merits of the proposed method. Particularly remarkable are the improvements obtained on clinical outcome prediction tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gIhVwxcocC", "reviewer": "Reviewer_PcWA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a cross-model framework to learn molecular embeddings jointly from the chemical structure and images of drug-treated cells. Using four contrastive losses to pre-train a model on a high-throughput imaging dataset, the paper demonstrated improved performance on three downstream tasks, including retrieving chemical structures or images from the other channel, predicting clinical outcomes, and predicting molecular properties. \n", "summary_of_the_review": "I would not recommend this paper for publication in ICLR due to my concern about the limited novelty and tech contribution. It is a nice piece of work that I enjoyed reading through. ", "strength_and_weaknesses": "Strength:\n- The work tackles the important problem of learning molecular representations by bringing high-throughput imaging data.\n- Performance improvement of the pre-training strategy is demonstrated on three different downstream tasks compared with SOTA.\n- An interesting case study is reported to find molecules that have similar effects as single gene overexpression.\n\nWeakness\n- While I am no expert on cross-modal learning, it seems the framework just embraces several state-of-the-art ideas in contrastive learning without novel technical contributions.\n- No ablation studies for combinations of contrastive losses\n- Given the application nature of this work, related work should discuss the state of the arts in learning embedding from cell images. Particularly, the published workSanchez-Fernandez, Ana, et al. \"Contrastive learning of image-and structure-based representations in drug discovery.\" ICLR2022 Machine Learning for Drug Discovery. 2022.\nseems conceptually similar to the proposed work.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "bEevtVC6Wo", "reviewer": "Reviewer_4Jez", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an approach to co-learn representations from molecular graphs and fluorescence microscopy images.", "summary_of_the_review": "The main idea of the work is not novel, there is a fundamental problem with the approach, i.e. that microscopy images cannot capture all changes in molecular structure, and the experiments are insufficient due to lacking baselines. The choice of the loss function with many terms is particularly problematic for learning, but this is not ablated. ", "strength_and_weaknesses": "Strengths:\n- A relevant problem is approached. The retrieval-task between cellular images and molecular graphs is appealing and significant. \n- The paper is clearly written and structured; Figures are informative; Notation follows ML standards\n- The authors took care of splitting the data appropriately since multiple  images correspond to one molecular graph\n\nWeaknesses:\n- The approach is not novel, but has been suggested before. \n- The presented results are not relevant due to missing embedding into related work, missing informative baselines and incorrect scope. \n- The presented work has several severe technical errors. There is also a fundamental problem in the approach (see below).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ENfzC36LDfK", "reviewer": "Reviewer_mr25", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present MIGA, a method for combining graphs of molecules and corresponding high-content cell images into a related embedding space. This learned embedding space can then be used for several tasks of clinical relevance, namely cell image retrieval, predicting clinical outcomes of drugs, and molecular property prediction. The authors evaluate MIGA against several reasonable baselines in all areas.", "summary_of_the_review": "Overall, the paper is well-done and the approach is an intriguing way to combine cell images and molecular structure. The experiments are very well-done and thought out. My hesitations about giving a more confident recommendation to accept are that the improvements, while consistent, are modest, and I am concerned that there is much other work out there that I am not aware of, given how many companies and researchers are working in this area.", "strength_and_weaknesses": "Strengths:\n- The method is well-reasoned and is intriguing in this context, in what is a very challenging problem but of great relevance for the biomedical community.\n- MIGA is applicable to several important real-world applications, namely clinical trial outcome and molecular property prediction.\n- The authors compare to an impressive number of baselines, showing a consistent advantage of their method.\n- Related, the authors have thoroughly considered proper metrics and ablation studies to evaluate MIGA.\n\nWeaknesses:\n- I am concerned that there may be other works in this area that are not considered. There are entire companies, such as Recursion, that work in this area of deep learning-based high-content image screens for drug design. I do not have a sense of how significant of an advance is their method in comparison to such work.\n- The performance increase is modest, which is expected given the difficulty of the problem. I believe it is worthwhile to publish still, given the consistency of the improvement for such an important problem, but it is still a weakness that it does not yield a more significant improvement.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject