Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Motivated by the genetic perturbations problem in biology, the authors reformulate this problem as a batched regret minimization problem with a large action space.  Specifically, they assume the feedback is noiseless and prior-known a potential function class that outputs the score of a given action (a perturbation) but this function class can be misspecified. 

Based on this setting, they propose the Optimistic Arm Elimination (OAE) framework, which can be adapted to various optimistic predictors. Under this framework, they give a theoretical regret guarantee in terms of the Eluder dimension of the function, the misspecification level, and the batch size. Then they test their framework on various experiments by choosing the regression model as deep neural net and get SOTA in 3 of 4 datasets.
strength_and_weaknesses: Strong: This is a well-motivated real-world problem. The author gives a meaningful formulation of this problem and shows promising experimental results.

Weaknesses:
1\ The proposed framework in the main paper, although high-level and flexible, seems just a very standard optimism-based framework that incorporates various existing heuristic methods  (Section 4.1, 4.2)

2\ Under the noiseless assumption, the proposed predictor in Eqn.(1) and the corresponding Theorem 4.1 seems not surprising to me -- If there is no misspecification then this is just a standard passive learning problem. If there is some misspecification, then it is linear in misspecification. Also this predictor is not parameter-free because you need to choose tolerance $\gamma$ based on misspecifcation.
clarity,_quality,_novelty_and_reproducibility: It is well-written. But I cannot find the reference for citations in 5.0.1 (those datasets)
summary_of_the_review: While I believe it is practical and meaningful based on those experimental results. It is a bit unclear to me whether its novelty on theoretical contributions or algorithm design contributions. I am not experienced in doing experiments so it is hard to judge the overall contributions.

Also, given this is an ml for a science paper. I am willing to raise my score if people familiar with this area endorse it.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors investigate the problem of predicting how to perturb cells via genetic manipulations in order to shape their phenotype in a desired way. Their approach to this problem is to rely on a novel form of batch query bandit optimization, their Optimistic Arm Elimination (OAE) principle. They use their OAE to show that they can often plan effective experiments in fewer tries than baseline methods.

strength_and_weaknesses: 

Summary
The authors investigate the problem of predicting how to perturb cells via genetic manipulations in order to shape their phenotype in a desired way. Their approach to this problem is to rely on a novel form of batch query bandit optimization, their Optimistic Arm Elimination (OAE) principle. They use their OAE to show that they can often plan effective experiments in fewer tries than baseline methods.


Strengths
- The problem setting is extremely interesting, and I believe holds huge potential for drug discovery. Also, this is a problem that I believe has been under studied because it is so difficult to model.
- The authors test their method on a variety of datasets.

Weaknesses
- I'm not sure what the purpose of Figure 1 is. It's showing regression loss of different models trained on a few cell lines. The model hasn't converged. There's no test performance. What should we take from this.
- What motivated the model designs? The authors switch between NN 1500-300 to NN100-10 to NN10-5 without much explanation. What's going on here?
- Figure 3 is above figure 2 — confusing.
- Results are extremely sparse and confusing. MeanOpt sometimes requires fewer batches than random or greedy and sometimes doesn't. On the GeneDisco experiments it looks like the author's models work the best on 3/4 conditions but I have no idea of effect size.
- The authors acknowledge that they assume noiseless responses and that they will focus on noisy responses in the future. For biological applications -- and especially for any kind of sequencing -- I'm worried that the noiseless assumption is overly optimistic even when operating on populations. I'd like to hear more from the authors on why an assumption of no noise is OK here.

Minor

- The citations in text do not always appear correctly (I think you should be using \citep sometimes when you are not).
- The model names are very confusing. A figure might help? Also easier to grok names? They're all the same type of model with different widths (right?), so something like narrow, baseline, wide might help.
clarity,_quality,_novelty_and_reproducibility: The introduction is clear but I found that the method section was too long and the results too short for my tastes. This is subjective, of course. Also the figures were too small and difficult to interpret, which was a bigger issue.

The method seems novel although i am not an expert in the area. The results appear to be reproducible.
summary_of_the_review: The problem setting is incredibly interesting and exciting but I thought the paper was difficult to read and the results not all that compelling. I am also worried that the key assumption of no-noise is fundamentally flawed for biological data, so I'm curious what the authors and other reviewers think about that. I think there's great potential here, however, as it stands, I am leaning towards reject but still open to changing my review during the discussion period.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a novel bandit / active learning method tailored to the problem of single-gene interventions in cell biology experiments, where we can have batches of parallel interventions and only a few rounds of active learning are feasible. The new bandit method (Optimistic Arm Elimination or OAE) is based on either an existing uncertainty estimator or a novel optimization objective for the reward predictor that trades-off between fitting the data and maximizing the predictions. A diversity regularizer can be added to regularize each batch to be somewhat diverse. A large set of experiments to compare the different variants under study is performed, along with one comparative experiment (on 4 datasets) on the GeneDisco benchmark (using the public implementation of GeneDisco).

strength_and_weaknesses: The paper is generally well-written (a few minor English errors should be easy to fix) and clear. The proposed method appears novel (although inspired by much previous work, and an extensive set of relevant papers are cited) and the results appear promising. The paper also includes a bandit theoretical results bounding the regret.

It is clear that the authors have thought carefully about many issues arising in the context of their targeted application (such as diversity within each batch, how to evaluate performance).

The main weakness is that there should have been more comparisons against existing methods rather than against a single strawman (the GeneDisco public implementation). 

The paper focuses on the single-intervention case and the algorithms would not scale to compositions of interventions (e.g. multi-gene interventions) because the different actions must be enumerated and scored separately. See the recent work on GFlowNets (NeurIPS 2021, ICML 2022) as a possible way to train a policy to generate actions in a way that favors a diversity of candidates in each batch, as an alternative to explicit screening of all the possible actions.

I am concerned with the use of the inf-norm in 4.3 because a hugely diverse set of \tilde{y}* could minimize it.

I would like to see something added to explain how to select lambda_reg (which controls the trade-off between fitting the data and maximizing predictions, i.e., the weight uncertainty in the acquisition function) and how to do it without cheating (e.g. to get the results in Fig. 5), i.e., without using the downstream results of the active learning iterations.

More references should be added regarding the related work in uncertainty estimation without an explicit Gaussian posterior (such as available with GPs), i.e., with neural networks. See e.g. DEUP (Lahlou et al 2021).
clarity,_quality,_novelty_and_reproducibility: High marks on clarity, quality and novelty (see above). I am not sure if the code is or will be shared. Please clarify.

summary_of_the_review: This paper is clear and describes a strong, novel and well-thought out work on bandit methods in cases where very few rounds of batched interventions are possible. It could be stronger with more points of comparison, i.e., against more of the existing state-of-the-art methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors define new methods to select batches of perturbations of cells that are most likely to inform and optimize on a phenotypic metric of interest.
strength_and_weaknesses: Very interesting problem, and not a lot of others in this space–great to define this problem concretely!

A wide array of related work is outlined and organized, not only with section headings, but also citations.

My difficulty with theory-heavy biology paper lies in statements like this: “the learner is required to query a batch of ..arms…and observe noiseless responses…In the setting of neural perturbation experiments the responses are the average of many expression values across a population of cells, and thus it is safe to assume the observed response is almost noiseless.” As someone who has analyzed dozens of these datasets, this is most certainly not true. In real world “shRNA and CRISPR” experiments, there is always variability, minimally in knock-down or edit level, phenotypic response, experimental design, etc. Though a population of cells is sampled, there will always be variability. Since you have datasets, can you characterize the experimental noise of a given knockout? If you don’t have this data, you cannot make this assumption.

To what extent can the uncertainty estimates for your model be used? How were they calibrated, and how much poor calibration can you tolerate? For proper uncertainty estimates, why not Gaussian Processes as a baseline, or maybe even Bayesian Neural Networks? Why are only Ensemble methods described in Section 4.1.2 as the only method to create uncertainty, but then Bayesian Neural Networks are used in Section 5.0.1? Why were those not described in Section 4?

In Figure 1, and that associated analysis, to what extent do we care about cross-cell-line variability? I think showing this generalization is useful. However, how much do we generalize across perturbations?

The use of a VAE to represent sequences is concerning to me. Why don’t just input raw data into a neural network? By using the VAE representations, you are implicity assuming you can reasonably represent all variability in expression. If a series of genes, or gene networks, were removed during training of the VAE, and projected with a learned model, how are they represented? If these KOs are incorporated into both the VAE and neural net training, can you still generalize to those predictions?

clarity,_quality,_novelty_and_reproducibility: First sentence of the abstract is not clear to the average ML reader. What is CAR-T, CAR-NK, CAR-NKT? As a trained ML bioinformaticist, I don’t even know the difference between those three.

Again for the average ML reader, what is shRNA or CRISPR? What is a genetic knockout? There are no references in the first paragraph of the introduction for the reader to continue reading from.

Section 1 - “​​Our experimental evaluation covers both neurally realizable and not neurally realizable function landscapes.” What does this mean?

Section 2 - “We provide guarantees for the no noise setting we study based on the Eluder dimension…” What does this mean?

Section 5 - Can we have some reference for the cell lines described VCAP, HA1E, MCF7, A375?

In Figure 2 - Where are the exact experimental details of the figure legends for MeanOpt, Greedy, and Lambda0.XX- ? It is unclear how these abbreviations tie into the methods described, and the reader should not have to hunt through the supplement to look this up.

In Figure 5, what is the variability around those estimates?

It is unclear how the conclusions described at the bottom of page 8 relates to Figure 4. For example, what does “thus indicating this function class is too far from the true responses values for A375” mean? What function class are you referring to? Though there are scores of figures and descriptions in the appendix, the important points to convince me, the reader, should be able to be summarized in the main work.

The metric you are optimizing differs for the two datasets, which makes comparison difficult.

summary_of_the_review: I think that the problem the authors are trying to solve is very interesting, and I applaud them on their work. However, I found that this paper lacked focus and refinement–important points should fit into the main text. Moreover, I found there to be a number of arbitrary design decisions with VAEs and uncertainty estimate models that detracted from the core contributions stated by the author.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The papers considers designing genetic modifications to cells to engineer their phenotype.  Evaluating genetic modifications is time-consuming and expensive. The paper develops a iterative batch method for exploring the space of modifications.  This method is based on the proposed Optimistic Arm Elimination principle. The paper presents an analysis of its convergence in terms of the Eluder dimension of the function mapping modifications to phenotypes.  The method is evaluated on an array of interesting datasets.

Strengths
- The application domain is important and interesting
- The paper develops a scalable and tractable way to use the principle of optimism (common in bandits) within the context of neural network approximations 
- The proposed method is accompanied by a non-trivial regret guarantee
- The evaluation is performed on a variety of interesting datasets
- The paper is clearly written overall

Weaknesses
- Analysis assumes noiseless responses
- Some minor opportunities to improve clarity and clarify experiments
justification_for_why_not_higher_score: - Concerns about the appropriateness of a zero-noise assumption, both in the application domain considered, and in other applications
justification_for_why_not_lower_score: - Very interesting problem & experimental evaluation
- High-quality analysis with a well-justified method
Decision: Accept: notable-top-25%