Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors propose a new model called chemCPA to study the perturbational effects of unseen drugs by incorporating knowledge about the drug's molecular structure.   They introduce new encoder-decoder architecture and benchmark against other molecule encoding networks. Also, the authors combine this model with a transfer learning scheme that improves generalization by utilizing existing HTS bulk RNA-seq data. 
strengths_and_weaknesses: The authors propose a model for predicting cellular gene expression responses for unseen drug perturbations by encoding the drugs’ molecular structures. The approach is well motivated and is described clearly. Authors have demonstrated that the pretraining on the abundance of bulk HTS datasets improves generalization. The chemCPA has several advantages like a latent space disentanglement and the induced interpretability of the perturbations in the context of cell state and covariates, incorporating molecular structure over regular encoders. 

One concern is that the performance of chemCPA models on DEGs is worse compared to all genes. Also, it may be beneficial to benchmark chemCPA against CPA or scGen in less complicated scenarios with a few perturbations. 

On line 60, computational perturbation auto-encoder (CPA) should be changed to Compositional perturbation autoencoder
questions: 1) I am not entirely clear on how subsetting of 977 genes is done via ensemble gene annotations?
2) have the authors looked into other datasets mentioned in the Compositional perturbation autoencoder paper?
limitations: Limitations are scattered across multiple sections. It may be much more helpful to assemble them in the discussion section for improved readability
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors describe a method for predicting the gene expression response of a cell line to drug treatment. They make use of an autoencoder framework, encoding the gene expression profile using a dense neural network layer. They use adversarial losses to discourage the model from encoding the cell line and drug treatment in the latent space. Finally, they derive features from the molecular structure of the drug using various approaches. Embeddings of the expression profile, cell line, and drug molecule are decoded with dense neural network layers and compared to the observed gene expression from the experiment by a loss function.

Aug 9 - I increased my ratings after reading the authors' revision and rebuttal.
strengths_and_weaknesses: Predicting cell responses to drugs is an important problem.

I don’t understand the evaluation. Why is the term “counterfactual” prediction used? It sounds like it refers to model encoding and decoding with DMSO as the molecule, but I don’t understand why this would be relevant. Are the authors evaluating performance on other molecules than DMSO? I’m confused by the R2 metric, which seems to be computed across genes for each sample, and the authors use of the term “disentanglement”. The authors describe Table 1 as if it evaluates disentanglement, but the metric is R2. What are the drug and cell line columns of Table 1 showing? In the case of DEGs, do the authors consider distinct DEGs for each molecule?

These ideas have been published previously. The authors don’t distinguish their method, and they don’t compare to previously published methods. E.g. see Lotfollahi, M. et al. Learning interpretable cellular responses to complex perturbations in high-throughput screens. bioRxiv 2021.04.14.439903 (2021) doi:10.1101/2021.04.14.439903. https://www.biorxiv.org/content/10.1101/2021.04.14.439903v2
questions: Line 90, Gaussian likelihood is inappropriate for RNA-seq.
Line 141, L1000 isn’t bulk RNA-seq, it’s a platform called NanoString.
Figure 3, in the caption, why are latent embeddings “identical” for molecules that target the same pathway?
Table 4, caption incorrectly describes table.
limitations: Yes, limitations have been addressed.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper suggests a new model (coined chemCPA) that predicts the transcriptomic perturbation of drugs on specific cell lines.
One of the highlights is that chemPCA uses some transfer learning approach to predict those perturbation at single cell level (for data is generally noisy and not widely available) while leveraging the vast amount of available bulk RNA-seq screens, even in the setting where the available set of genes are different between the source and the target domains.
Building on the previously published CPA model, chemPCA relies on an  a similar autoencoder architecture but uses existing pre-trained structure embeddings of the drugs to allow generalization to unseen compounds (or unseen combinations of seen compounds).
After introducing and briefly justifying the chemCPA architecture, the authors evaluate the performance of their method in an extensive setting, benchmarking first the choice of various chemical encoders, then validating their transfer learning (from bulk RNA-seq to the scRNA-seq domain) and finally the generalization performance to unseen drug compounds.
Last, the authors suggest an uncertainty measure on the drug embedding which experimentally seems to correlate well with how well chemPCA generalizes to unseen compounds.


09/08/2022 : after carefully reviewing the author's response to all reviewers, as well as the amended manuscript, I am updating my rating of the presentation from 2 to 3 and my overall rating of the paper from 5 to 8.
strengths_and_weaknesses: Given the amount of efforts and resources spent on screening, in vitro, chemical compounds in the field of drug design, developing efficient and reliable in silico models that are able to predict the perturbations induced by such compounds on a transcriptomic level is of utmost importance.
While this problem has been the object of a lot of research over the last decades, multiple limitations still makes it an open problem.
Although I am not an expert of this subfield, I think that addressing specifically the following questions: 1) how to generalize predictions to unseen drugs (or combination of) and 2) if it is possible to transfer information from the very abundant and less noisy (but "lower resolution") bulk HTS data to the emerging (hence sparsely available for the moment) single cell HTS, is key to the development of the field.
While chemPCA admittedly inspires from already existing ideas, their combination enables novel applications and use cases. As such, I consider the novelty of this work to be satisfactory and its significance high.

I have some concerns about the clarity of the manuscript.
The experimental results of this paper are more extensive than the average NeurIPS submission with settings that require a lot of explanations.
I have the impression that this work would benefit from a longer format to improve (e.g. a bioinformatics or computational biology journal).
For a ML venue such as NeurIPS, I would have appreciated a bit more insights on some of the choices / assumptions made (see the Questions below for more details).
Some important details seem to be completely eluded. For instance, there is a mention of an "architecture surgery" to match the larger dimension of the extended set of genes of the target domain.
However, no real explanation is provided as to how this is executed. Is it related to the scArches paper [Lotfollahi et al. '22]? If yes an explicit reference to it would be welcome and the manuscript would benefit from more explanation (at least in the appendices).
Also, on a more technical level, many of the abbreviations (CPA, MoA, DMSO etc.) used are not defined properly (not defined at all, or after they were first used). It is a simple detail but fixing this would improve the clarity.
In Figure 3, the test compounds are labelled, but it is very difficult to know which label corresponds to which point. That makes it hard to even see if the test compounds are actually clustered at all with other drugs affecting the same pathway.

I also have concerns about the experimental results.
I find the setting and design of the experiments to be very thorough and interesting.
My main concern is that it is mostly a very elaborate ablation study where the impact of the choice of the drug encoder, the impact of pretraining are studied (also in terms of generalization to unseen drugs).
However, there is no comparison to any competing method or baseline (other than using the unperturbed measurements).
I understand that chemCPA has a quite unique positioning and that there are likely no other methods that predict perturbations of unseen drugs on single cells.
However, I think it would be relevant to test chemCPA on less ambitious settings where it can be compared to existing methods (e.g. predicting the perturbation of unseen pairs of cell lines and seen drugs).
Significantly improving over the SotA on such tasks would not be necessary, but showing that the results are not bad in such settings would, in my opinion, make the manuscript stronger.

I am well aware that my remarks go in the direction of adding even more content to an already very dense paper. Maybe some of the experimental details and results could be relegated to the appendices.

Overall, I have no doubt that the work underlying this submission is novel and significant enough to be the object of a relatively high impact publication. However, I think that in its current form, the paper is not reflecting the high quality and potential of this work.
If the authors answer my concerns convincingly and have a plan to update their manuscript accordingly, I could very well be convinced to improve my rating.

questions: In addition to comments on the points I raised in the previous section, here is a list of more specific questions I would love to see the authors answer:
 1) While it may be a very reasonable idea, I am under the impression that assuming that basal states and cell lines can be completely disentangled is somewhat of a strong statement. Do you have any argument or insight that support this choice of modelling? Or is it an approximation that is necessary to make the problem tractable? Some further comments would be very appreciated (and would, I think strengthen the manuscript if included).
 2) If I understood correctly (but I admittedly did not dig very deep into [Lotfollahi et al. '21]), the current setting is limited to predicting the perturbation induced in a fixed, known in advance, set of cell lines. Do you think this work could be extended to generalize to different cell lines, or, perhaps more interestingly to predicting perturbations in cells that are not coming from a cell line (e.g. from a biopsy)? Do you have insights on whether it would be completely unrealistic to use a vector of gene expression levels before perturbation as an input rather than an encoding of the cell line?
 3) A Gaussian likelihood reconstruction loss is used (and the data is log-normalized and standardized as a preprocessing step). Why not using a (Zero-Inflated?)Negative Binomial model that should be a better fit to such count data? Is it something that you tried or considered?
 4) There is no discussion on the choice of this additive model for the perturbations. I could imagine that the non-linearities induced by the encoder and decoder architectures mean that such an additive model may very well not be limiting the expressivity of chemCPA. It would be nice to have some comments about it, though.
 5) Related to the last question but more specific, reducing the dosage attributes to a single scalar value applied to the drug embedding vector intuitively feels like a very strong underlying assumption. Do you have any comment on that?
 6) A gradient penalty term (inspired from the Improved WGAN paper) is added to the adversarial loss. First, in [Gulrajani et al. '17], it was penalizing the gradients to have norm close to 1 (not a penalty on the norm directly). Second, having gradient norms equal to 1 was a proxy to having a 1-Lipschitz critic / discriminator because that is what made optimizing the WGAN objective equivalent to minimizing the Wasserstein distance between real and generated samples. Here, I don't see the connection at all. Any comment on the choice of this regularizer?

limitations: No discussion about the societal impact was included in the manuscript. But like the authors, I also do not foresee any impact so I don't see it as a problem.
In general, as written in the previous sections, my sense is that the manuscript does not comment enough on the modelling choices that were made. As such, I find the limitations of their work not to be very carefully addressed. Please refer to the other sections of this review for more details.
review_title: null

Meta-review: metareview: All four reviewers liked aspects of the paper with one still on the fence. All reviewers appreciated the authors' feedback to the their comments both in the discussion and the extensive updates of the paper.

Accept is recommended.
confidence: Certain
Decision: Accept