Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work presents a new probabilistic algorithm ("DiscoBAX") for subset selection that aims to approximately optimize phenotype movement in genomic intervention and can be useful in drug discovery tasks according to the authors. The method identifies a set of interventions whose elements will trigger maximum expected change in the disease outcome with respect to for some noise distribution. Performance is assessed based on synthetic data as well as public benchmarking data, and the algorithm is shown to outperform alternative approaches.


strength_and_weaknesses: Strengths:
+ probabilistic approach is neat for uncertainty characterization
+ practical combination of task, method derivation, and algorithm implementation 
+ Openly licensed source code for the algorithms available

Weaknesses:
- Figure numberings are broken, text cites figure up to Fig.5, manuscript only has 3 figures, refs to Figs 3-4 are missing; not clear how to follow
- The problem motivation is hypothetical; it is not clear that "maximizing change" is what one would like to achieve in genomic drug discovery; on the other hand the algorithm is of interest regardless but perhaps best evaluated on its own right as a target optimization task.
- I am not sure if potential overfitting is sufficiently addressed in this work; explanation on this part could be strengthened 
clarity,_quality,_novelty_and_reproducibility: Quality:
- Overall the paper is well written; the problem motivation and validation part (esp. overfitting; see comments) could be strengthened. Figure numbering is broken.

Clarity:
- Text is easy to read and follow
- Some more intuitive descriptions of the algorithm could be added
- Figure numbering is broken.

Originality:
- The work provides a new solution to a previously established problem and claims performance gains
- The combination of problem and algorithmic solution seems to be new 

Novelty And Reproducibility:
- I did not try to replicate the work but code is well organized and openly licensed, and seems robust. Some more guidance in the README landing page would be warranted.

summary_of_the_review: The work provides an interesting approach to an optimization problem that is initially motivated by certain genomic drug discovery tasks. Overall the reporting is clear and appears reproducible. The algorithmic innovation is in place but focuses on a relatively specific task definition where the applied motivation remains questionable, and it is not clear whether overfitting has been sufficiently controlled as the algorithm appears rather flexible. The paper has figure numbering problems.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper considers the problem of designing an experiment where there are two stages: in the first (in vitro) stage our task is to (efficiently) design an experiment that will have good chance of success in the second (in vivo) stage. The authors formalize this elegantly in equation (3).

The proposed solution is discoBAX: an algorithm that actively searches the in-vitro function for a set of optimal yet diverse points that have the best shot at having a successful outcome in the in vivo stage. 

The paper explores discoBAX on a synthetic and real-world dataset.


strength_and_weaknesses: Strengths
--
After spending some time with the paper, I came to appreciate the formalisation of the setting that is being worked in. I particularly enjoyed section 3 which laid out the problem. I appreciate the authors use of the appendices to keep long proofs out of the paper (the submodular proof for example) and also the clear presentation of the algorithms used. Posting the code anonymously gives me some confidence that the work is sound. 

Weaknesses
--
the paper fails to deliver what is promised at the start of the experimental section - an investigation of discoBAX's sample efficiency, sensitivity to hyper-parameter settings, and comparison with the top-intervention method. As a result I feel that the paper is really rather unfinished! I am keen to understand more about discoBAX and am disappointed that the investigation is not deeper. 

The 'related work' section at the back does not cover BAX - which appear to be prior art? this section feels like filler, I would have preferred the space be used to investigate discoBAX further. 

I did not find a satisfactory discussion (or empirical evaluation) of the use of the relu'd score function, relative to the original objective. I get that this makes the optimization submodular, but have we lost anything? In which cases does this cause a problem?

It took me some time to work out what was going on - I think more discussion around Figure 1 is needed, and perhaps some commentary on how this is connected to the invitro/in-vivo setting. I liked the clarity of eq 3, but this felt like a long time coming, and would have been better placed at the start of the paper perhaps?

clarity,_quality,_novelty_and_reproducibility: I have to give the authors top marks for reproducibility since the code is available. 

A problem with the paper is that it is hard to me to assess novelty. Although the authors do lay out their contributions at the beginning, it's not clear to me whether this paper or another is the source of the technical inventions. For example, the submodularity of the relu'd score function. 

A minor (clarity) point, but the figures are poor quality. I cannot read the axis labels when printed out. Unfortunately this gives the impression of a rushed paper. 
summary_of_the_review: A really interesting setting to be working in, and what seems like a good idea, but a lack of empirical investigation into the method lets the paper down. Clarity is a problem and it's difficult to assess which parts of the work are a novelty. I could change my "technical novelty' score if the authors could clarify.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper is concerned with the iterative selection of an optimal set of targets for genetic interventions based on a scalar readout. The methods section described a particular instance of BAX, that models the singularity of the biological problem that is considered. A few aspects that makes it different from the existing InfoBAX approach is (1) the subset maximization target function and (2) a practical uncertainty model with BNN instead of GPs. Then, the paper provides a set of experiments on synthetic data, and on the openly available GeneDisco benchmark. 
strength_and_weaknesses: Strength:
(1) Novelty *in its application*: this is one of the (relatively) few papers that tackle this important scientific problem (other example I could find is https://arxiv.org/pdf/2207.12805.pdf).
(2) The paper shows improved results in the recent GeneDisco benchmark, for a few of the datasets.

Weaknesses:
(1) My main issue is that I am having a hard time understanding the *technical* novelty of the paper. A lot of the concepts explained in Section 4 seems to belong more to a background section, as they are not contributions of this particular work, but from the BAX paper. As far as I understand, the contributions are (a) working with sets, and performing greedy set optimization and (b) using BNN for uncertainty quantification. However, in the results section, I don't see a clear explanation that those changes are improving the performance. It seems that (a) improves diversity? (if so, it should be made clear), but (b) has no comparison to different flavors of BNNs, GPs etc..
(2) there is no theoretical analysis on how the (1 - 1/e) approximation error propagates through the BAX procedure. Then, it is not correct for the author to write "theoretical guarantees on the optimality of the approach". Similarly, the authors write that DiscoBAX is "sample-efficient", but I have seen no proofs of sample efficiency, and the experimental results are not necessarily strong enough to claim this.
(3) There is some improvement over BAX, but only in a few datasets (2 / 4). 
(4) There should be a comparison to other BNN flavors, and at the very least a discussion of uncertainty modeling w/ neural networks

clarity,_quality,_novelty_and_reproducibility: The paper does not make its contributions clear, and I do think that it is currently an issue. Otherwise, the paper reads well. The application of BAX to genomics is novel (but the practical implications for the field of genetics are rather unclear), but the improvement over BAX is not clear (as the paper is written now). 
summary_of_the_review: This paper provides an interesting application of BAX, and investigate a practical variant to batch learning of genetic perturbations. Beyond this, the technical contributions may be considered weak, and the empirical results are not exactly strong either (please refer to my previous sections for actionable items). Regarding the experimental results, I completely understand that it might be caused by noise in some datasets (e.g., COVID). In this case, I do think that the authors should carefully present ablation studies so that readers can understand what features of DiscoBAX helps in increase performance compared to BAX (again, see above for suggestions). In the light of this, I am currently leaning towards rejection of this manuscript, but very much looking forward to discussions with authors, and other reviewers.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper considers the problem of finding a `diverse` set of genomic interventions that maximize a phenotype of interest. Formally, the problem can be modeled as optimizing expensive black-box functions over sets of inputs from a given design space. Bayesian Algorithm eXecution (BAX) is a recently proposed framework that allows estimating required properties of black-box functions using information gain based input acquisition strategies. The paper provides a BAX (Bayesian `Algorithm` eXecution) style approach to solve the problem where the key idea is to consider the subset maximization of a chosen score function as the `Algorithm`. At each iteration, an Expected Information Gain acquisition objective is optimized to select the next points for evaluation. This objective is parameterized by multiple sample outputs of the subset maximization algorithm. Experiments are performed on synthetic and GeneDisco benchmarks.
strength_and_weaknesses: - The problem considered in the paper is important with real-world implications. I found the biological motivation of the problem well-written with good intuitions for a non-domain person.

- The overall idea of using BAX style approach for this problem setting is fairly interesting and novel. 

However, I have few questions  to understand some of the details better and some suggestions that will hopefully improve the paper's contributions:

- It is mentioned in the paper that the discrepancy between the disease outcome $f_{out}$ and intermediate phenotype $f_{ip}$ is captured by the noise distribution $\eta$. It is motivated as an important distinguishing factor between the problem setting of this paper compared to that of existing approaches like Bayesian optimization. However, there is little clear description about estimating this quantity or principles behind choosing a certain distribution. A short remark is mentioned after observation 1 in few lines but that seems limited for such an important motivation of the problem setting. Please describe the concrete implementation/algorithmic details of the noise distribution and how it is estimated in the experiments. 

- The proof of submodularity of the score function in appendix A is missing some description. For example, $dis(g, \eta)$ is not defined. Please expand the proof and explain the reasoning behind the inequalities. If it is `straightforward` (as mentioned in the first line of the proof), can it be considered a major contribution of the paper? 

- The score function is chosen to be the best value in a set (averaged over noise distribution eta). This seems like an optimistic choice. For the algorithm to find good robust points, should we not consider the worst value in the set as the score function?

- Please add a description of the computational complexity of the proposed approach. Since multiple instances of the subset maximization algorithm needs to be run in each iteration for estimating the expected information gain quantity, the computational complexity can be quite high. A wall-clock time comparison of the proposed approach with existing baselines will be very useful, especially since the top-k recall gap is relatively small on most of the benchmarks (other than Leukemia/NK cells).

- As mentioned in section 5.2, batch size of 32 points are selected for evaluation in each acquisition cycle. How is the batch of points selected? Is it accomplished by greedy optimization of the EIG acquisition function? If yes, what are the accuracy losses of doing this greedy optimization. 

- The choice of UCB as Bayesian Optimization (BO) baseline strategy seems surprising. There is a large literature on Bayesian optimization algorithms (please see [1-4] and references therein) for the batch evaluation setting that should be the right comparison here. Something like qEI (q-Expected Improvement) is easy to setup and implemented in popular packages like BoTorch [1] (https://botorch.org/tutorials/). The points suggested by qEI are also known to find highly diverse points in the context of batch optimization. Please consider improving this comparison (by including qEI for instance) because BO algorithms are directly applicable and the most relevant baseline in the setting. 

References 


[1] Balandat, Maximilian, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G. Wilson, and Eytan Bakshy. "BoTorch: a framework for efficient Monte-Carlo Bayesian optimization." Advances in neural information processing systems 33 (2020): 21524-21538.
González, J., Dai, Z., Hennig, P., & Lawrence, N. (2016, May). Batch Bayesian optimization via local penalization. In Artificial intelligence and statistics (pp. 648-657). PMLR.

[2] Wu, J., & Frazier, P. (2016). The parallel knowledge gradient method for batch Bayesian optimization. Advances in neural information processing systems, 29.

[3] Azimi, J., Fern, A., & Fern, X. (2010). Batch bayesian optimization via simulation matching. Advances in Neural Information Processing Systems, 23.

[4] Gong, C., Peng, J., & Liu, Q. (2019, May). Quantile stein variational gradient descent for batch Bayesian optimization. In International Conference on Machine Learning (pp. 2347-2356). PMLR.


clarity,_quality,_novelty_and_reproducibility: The problem setting is clearly defined and the usage of BAX style approach is novel. It is commendable that source code is made available for easy reproducibility. 
summary_of_the_review: I find the overall approach to be quite interesting while addressing an important problem. However, some of the design choices and baseline comparisons require more work to improve the quality of the paper for an ICLR publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors in this work proposed a new method for defining the genetic interventions that could maximize the chance to direct a desirable phenotype in a complex biological system. The authors demonstrated their approach using synthetic as well as real world datasets to show good performance compared with benchmark. The overall concern is that the scope and the significance of the advancement of the work is limited. 



justification_for_why_not_higher_score: While it is very commendable that the authors tried to revise and improve the paper during discussion, the overall concerns from the reviewers are not sufficiently addressed, but the paper has well-grounded motivation and interesting ideas that should be noted. It would be very good for the authors to further improve based on the reviewers' comments for future submission.
justification_for_why_not_lower_score: NA
Decision: Reject