Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present MIGA, a method for combining graphs of molecules and corresponding high-content cell images into a related embedding space. This learned embedding space can then be used for several tasks of clinical relevance, namely cell image retrieval, predicting clinical outcomes of drugs, and molecular property prediction. The authors evaluate MIGA against several reasonable baselines in all areas.
strength_and_weaknesses: Strengths:
- The method is well-reasoned and is intriguing in this context, in what is a very challenging problem but of great relevance for the biomedical community.
- MIGA is applicable to several important real-world applications, namely clinical trial outcome and molecular property prediction.
- The authors compare to an impressive number of baselines, showing a consistent advantage of their method.
- Related, the authors have thoroughly considered proper metrics and ablation studies to evaluate MIGA.

Weaknesses:
- I am concerned that there may be other works in this area that are not considered. There are entire companies, such as Recursion, that work in this area of deep learning-based high-content image screens for drug design. I do not have a sense of how significant of an advance is their method in comparison to such work.
- The performance increase is modest, which is expected given the difficulty of the problem. I believe it is worthwhile to publish still, given the consistency of the improvement for such an important problem, but it is still a weakness that it does not yield a more significant improvement.
clarity,_quality,_novelty_and_reproducibility: Clear:
- Overall, the paper is well-written and easy to follow. The logic of the introduction, methods, and results flow well.
- The figures give helpful explanation of the context of the problem and the method and its applications.

Unclear:
- The cell images are in general very small throughout the paper. It is hard to perceive the details. For example, in the examples for image retrieval in Fig. 3, the individual cells are very small. The images of molecules are also small in these figures, though this is not as significant of an issue as the cell images.
-  A related issue is the contrast of the images being very low. Even after zooming in, I admit I cannot perceive much difference between the images of BRCA1 overexpression and TP53 overexpression. 

- The method seems reproducible.
- The method is significantly novel, to my knowledge.
summary_of_the_review: Overall, the paper is well-done and the approach is an intriguing way to combine cell images and molecular structure. The experiments are very well-done and thought out. My hesitations about giving a more confident recommendation to accept are that the improvements, while consistent, are modest, and I am concerned that there is much other work out there that I am not aware of, given how many companies and researchers are working in this area.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work utilizes the self-supervised learning paradigm to improve the data representations in Graph Neural Networks. Different from previous literature, the proposed method employs cross-modal correspondences between the Graphs of molecular structures and the associated (paired) cell microscopy images. The intuition behind combining these specific modalities is clear, drug molecules have the ability to cause perturbations (subtle morphological changes) to cells, e.g. their shapes, numbers, structures, and so on. To that end, the paper proposes a pretraining method called MIGA that processes these paired modalities, and aligns them in the feature space. MIGA is trained with three combined loss terms, which are inspired from existing literature on contrastive learning and generative modeling. Then, the learned data representations, stored in the form of neural network weights in the Graph and Image encoders, are evaluated on several downstream tasks. The chosen downstream tasks seem relevant to the domain, and are sensible overall. The reported results on these downstream tasks confirm the improvements obtained by pretraining by their method, MIGA, as opposed to the baselines from literature.
strength_and_weaknesses: 1- The proposed method, MIGA, utilizes the cross-modal information that exist naturally in microscopy images to improve graph data representations. While the underlying algorithms and losses to perform this cross-modal alignment are not novel, the idea of employing these heterogeneous modalities is inspired and novel. 
2- The paper is easy to understand, and the used language is clear. 
3- The experimental results are comprehensive, and confirm the merits of the proposed method. Particularly remarkable are the improvements obtained on clinical outcome prediction tasks.
clarity,_quality,_novelty_and_reproducibility: The paper is of high quality and is clearly written.
summary_of_the_review: Overall, I recommend accepting the paper. The method is novel, the writing is clear, and the experimental results are sound.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose an approach to co-learn representations from molecular graphs and fluorescence microscopy images.
strength_and_weaknesses: Strengths:
- A relevant problem is approached. The retrieval-task between cellular images and molecular graphs is appealing and significant. 
- The paper is clearly written and structured; Figures are informative; Notation follows ML standards
- The authors took care of splitting the data appropriately since multiple  images correspond to one molecular graph

Weaknesses:
- The approach is not novel, but has been suggested before. 
- The presented results are not relevant due to missing embedding into related work, missing informative baselines and incorrect scope. 
- The presented work has several severe technical errors. There is also a fundamental problem in the approach (see below).
clarity,_quality,_novelty_and_reproducibility: ### Clarity
The paper is clearly written and structured.

### Novelty 
The almost exact same approach has been presented in ref [1]. Even the identical dataset (cell painting) and contrastive learning objectives (InfoNCE, CLIP-like) have been used. The authors should propose a new method, approach and concept and state their novel contributions, e.g., at the end of the Introduction section.

### Quality
i) There is a fundamental problem with the approach and this is the focus on pre-training the graph neural network, i.e. molecule encoder, using cellular images. Many changes in molecular structure, do not lead to changes in cell morphology, such that the graph network is not required to code these changes in the learned representation. However, for downstream tasks exactly these structural changes might be relevant. Comparing to CLIP: CLIP focuses on pre-training the image-encoder (equivalent here: cellular image encoder) and not on the text-encoder (equivalent here: graph encoder) because for the text-encoder other pre-training tasks, such as masking/de-noising, are more appropriate. The authors should justify their focus on the graph neural network encoder. 

ii) It is unclear why multiple types of losses are necessary to train this architecture. For CLIP-like methods, the loss in Eq(3) is usually sufficient. Since the loss terms (Eq (9)) are not balanced against each other (except for Eq(8)), it is very likely that one of the loss terms dominates over the others, such that the training could completely neglect the image-modality. The ablation study (Section 4.3) does not investigate this at all. However, the bottom of Table 1 provides some insights: the strongest increase of performance arises from the masking loss, indicating that the cellular images are hardly used. Therefore, a pre-training strategy purely using the MGM loss (Eq 4,5) might be superior to cross-modal pretraining. Due to the missing error bars, it is unclear whether there is a difference between MIGA and “GIC+MGM” at all. The authors should justify their design choices and perform an ablation study to demonstrate that all four loss terms are necessary and need not be balanced against each other.

iii) Missing embedding into related work. The author's are unaware of closely related work ([1]) and also unaware that their suggested graph-image contrastive loss (Eq (3)) is identical to the CLIP algorithm. The authors focus on pre-training the graph neural network to encode molecules, but are unaware of related work on molecule encoders that perform much better on the MoleculeNet tasks (Table 3). 

iv) Informative baselines are missing. I quickly compared against a simplistic baseline, a fixed encoder (ECFP [7]) and linear probing and reached 77.05 on Tox21 and 69.43±0.0 on ToxCast. This baseline outperforms MIGA indicating that the graph neural network has not even learned to encode as well as an encoder that just checks for all substructures [7]. Furthermore, the best methods in the Tox21 data challenge reached an AUC of 0.845, with mostly descriptor-based approaches at that time, which means that the pre-training task for the GNNs is lacking. Comparing how much computational effort (energy and carbondioxide costs) it takes to pre-train the MIGA molecule encoder that in the end does not even perform better than an encoder that costs almost nothing (ECFP), makes it really difficult to justify the use of MIGA. Or in other words: the suggested approach to properly pre-train graph neural networks would mean that first a multi-million dollar experiment to collect the microscopy images has to be done. The authors should include informative and state-of-the-art baselines into their method comparisons. In table 2, it is unclear on which representation LR, RF and XGboost are running. The authors should use standard molecular fingerprints and descriptors as baselines in all studies. 

v) Presented performance metrics in many of the tables (e.g. Table 3 and 4) are results of single runs and therefore any performance differences can arise just by chance. For the ones with standard deviations it is unclear what they represent: re-runs on the same split or cross-validation? The authors should perform re-runs, or cross-validation, to obtain error bars and confidence intervals on all performance metrics.

### Reproducibility 
The dataset is publicly available, but needs strong pre-processing to be used for ML. The authors provide code as supplementary material, which makes the work likely to reproduce. One difficulty could be the dependency on the preprocessing of the Cellpainting dataset

References:
[1] Sanchez-Fernandez, A., Rumetshofer, E., Hochreiter, S., & Klambauer, G. (2022, March). Contrastive learning of image-and structure-based representations in drug discovery. In ICLR2022 Machine Learning for Drug Discovery.  
[2] Yang, K., Swanson, K., Jin, W., Coley, C., Eiden, P., Gao, H., ... & Barzilay, R. (2019). Analyzing learned molecular representations for property prediction. Journal of chemical information and modeling, 59(8), 3370-3388.  
[3] Ma, J., Sheridan, R. P., Liaw, A., Dahl, G. E., & Svetnik, V. (2015). Deep neural nets as a method for quantitative structure–activity relationships. Journal of chemical information and modeling, 55(2), 263-274.  
[4] Jiang, D., Wu, Z., Hsieh, C. Y., Chen, G., Liao, B., Wang, Z., ... & Hou, T. (2021). Could graph neural networks learn better molecular representation for drug discovery? A comparison study of descriptor-based and graph-based models. Journal of cheminformatics, 13(1), 1-23.  
[5] Wang, H., Kaddour, J., Liu, S., Tang, J., Kusner, M., Lasenby, J., & Liu, Q. (2022). Evaluating Self-Supervised Learning for Molecular Graph Embeddings. arXiv preprint arXiv:2206.08005.
[6] You, Y., Chen, T., Shen, Y., & Wang, Z. (2021, July). Graph contrastive learning automated. In International Conference on Machine Learning (pp. 12121-12132). PMLR.  
[7] Rogers, D., & Hahn, M. (2010). Extended-connectivity fingerprints. Journal of chemical information and modeling, 50(5), 742-754.  

summary_of_the_review: The main idea of the work is not novel, there is a fundamental problem with the approach, i.e. that microscopy images cannot capture all changes in molecular structure, and the experiments are insufficient due to lacking baselines. The choice of the loss function with many terms is particularly problematic for learning, but this is not ablated.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a cross-model framework to learn molecular embeddings jointly from the chemical structure and images of drug-treated cells. Using four contrastive losses to pre-train a model on a high-throughput imaging dataset, the paper demonstrated improved performance on three downstream tasks, including retrieving chemical structures or images from the other channel, predicting clinical outcomes, and predicting molecular properties. 

strength_and_weaknesses: Strength:
- The work tackles the important problem of learning molecular representations by bringing high-throughput imaging data.
- Performance improvement of the pre-training strategy is demonstrated on three different downstream tasks compared with SOTA.
- An interesting case study is reported to find molecules that have similar effects as single gene overexpression.

Weakness
- While I am no expert on cross-modal learning, it seems the framework just embraces several state-of-the-art ideas in contrastive learning without novel technical contributions.
- No ablation studies for combinations of contrastive losses
- Given the application nature of this work, related work should discuss the state of the arts in learning embedding from cell images. Particularly, the published workSanchez-Fernandez, Ana, et al. "Contrastive learning of image-and structure-based representations in drug discovery." ICLR2022 Machine Learning for Drug Discovery. 2022.
seems conceptually similar to the proposed work.

clarity,_quality,_novelty_and_reproducibility: Regarding clarity:
The paper is well written in general and can be improved by filling in some missing details:
- Parameters to generate masked context graph modeling in eq 4 and 5. It would also be helpful to clarify the cross-entropy used in eq 5.
- Eq 7: There should be k=1 instead of k in the summation?
- Details about the VAE architecture and training in Generative Graph-image matching are not provided.
- It would be helpful to discuss what features other than molecular embeddings are used in Clinical Outcome Prediction and how it fits into the pre-trained GNN.
 - It would be helpful, if possible, to provide details of cDNA and molecules used in the case study of zero-shot graph retrieval. Would be good to see Hit@10 of other SOTA as well. 

Regarding quality and originality:
The work is a nice application of representation learning in biology and drug discovery, albeit It seems the paper did not provide a new technical contribution. The idea of contrastive learning from chemical structures and cell images seems not new.  Performance improvement is properly demonstrated by evaluation experiments and case studies. 

summary_of_the_review: I would not recommend this paper for publication in ICLR due to my concern about the limited novelty and tech contribution. It is a nice piece of work that I enjoyed reading through.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a cross-modal contrastive learning method that combines a cross-modal contrastive loss with several uni-modal losses to improve the analysis of cell images.

A Strength of the paper is the well-motivated and relevant application and the  comprehensive experimental results, that show that the paper  improves over the SOTA, esp on clinical outcome prediction tasks. The reviewers also find the paper is well-written and well-presented.

The main weaknesses are that the paper has limited novelty in terms of cross-modal alignment methods. The method also combines a larger number of loss functions, making it quite over-engineered.

justification_for_why_not_higher_score: We did not reach a unanimous agreement on wether the positive or the negative arguments outweigh each other. Yet, as the authors have weakened their claim, the paper would require a significant re-write. Thus I now tend towards rejection on a very borderline paper.
Decision: Reject