Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a multi-objective Bayesian optimization approach for the molecules design problem. The proposed approach uses the hypernetwork-based GFlowNets as an acquisition function optimizer and uses a scalarization approach to combine the multiple objectives.

strength_and_weaknesses: Strengths: 
+ The paper presents an important scientific application. 
+ The paper presents some promising experimental results, though I have some reservations about the robustness of the results
+ The paper is easy to follow 
+ The paper addresses the multi-objective problem, which is relatively less studied in the context of molecules, but it is worth noting that it has been recently extensively studied in the general Bayesian optimization problem

Weaknesses

+ The proposed technique is a direct combination of existing techniques with no new substantial addition. Therefore, the technical contribution and novelty are weak
+ The paper uses the following statement “We assume that the oracle can be called as many times as necessary.” In Expensive settings, this is not usually true. 
+ The paper does not provide any time complexity analysis of the training. This is problematic because existing approaches are actually very fast, while gflownet is certainly much more expensive, so a time comparison and a discussion about complexity and tradeoffs are important.
+ The paper discusses the sampling of the scalars extensively, but later in experiments, it is mentioned that 5 evenly-spaced preference vectors are used. It is not clear how this works exactly. 
+ State of the art 
    - State-of-the-art methods in molecular optimization are not stated or compared to.  The following is considered SOTA work and covers a wide range of relevant methods and benchmarks that should be discussed for fairness. [1]
    - Most multi-objective BO papers are not mentioned nor compared to. In batch optimization the most efficient and high-performing methods are [2,3,4]. It is also misleading to state that no previous paper discussed diversity while [2] is a diversity-oriented method, and several single objective batch BO papers discussed diversity. 
    - The paper uses a scalarization technique where scalars are sampled from a distribution. This technique was previously proposed and used [5]. 
    - The Preference-based problem has been studied beyond the discussion that was mentioned in the paper. It is concerning to completely ignore principled existing work and claim it as a novelty. The following are some of the approaches, to name a few [6,7,8]

+ The experimental setup is weak and surprising:
- the paper states in the beginning that the evaluation was on several synthetic experiments and real-world experiments, but there are only two experiments. 
    - The paper uses three runs only to report the mean and standard error. BO papers report AT LEAST 10 runs usually, and most recent papers report 50 to 100 runs. I don’t think 3 runs can provide any statistical significance or deliver any conclusions about performance. In fact, even expensive deep learning models are usually tested with a higher number of runs. 
    - The diversity is not reported for some of the algorithms. If diversity is a metric applied to the Pareto front, why can’t it be applied to some of the approaches? 
    - The paper reports results for batch size 100 only. Batch BO papers usually evaluate several batch sizes. 
    - There is a total absence of many relevant baselines from molecular optimization, Bayesian optimization, and preference-based optimization. The paper is mainly experimental since the technical novelty is weak. Therefore, it needs to present a thorough experimental evaluation. 

[1] Maus, Natalie, et al. "Local Latent Space Bayesian Optimization over Structured Inputs." arXiv preprint arXiv:2201.11872 (2022).

[2] Konakovic Lukovic, Mina, Yunsheng Tian, and Wojciech Matusik. "Diversity-guided multi-objective bayesian optimization with batch evaluations." Advances in Neural Information Processing Systems 33 (2020): 17708-17720.

[3] Eric Bradford, Artur M Schweidtmann, and Alexei Lapkin. Efficient multiobjective optimization employing gaussian processes, spectral sampling and a genetic algorithm. Journal of global optimization, 71(2):407–438, 2018.

[4] Syrine Belakaria and Aryan Deshwal. Uncertainty-aware search framework for multi-objective bayesian optimization. In AAAI Conference on Artificial Intelligence (AAAI), 2020.

[5] Paria, Biswajit, Kirthevasan Kandasamy, and Barnabás Póczos. "A flexible framework for multi-objective bayesian optimization using random scalarizations." Uncertainty in Artificial Intelligence. PMLR, 2020.

[6] Abdolshah M, Shilton A, Rana S, Gupta S, Venkatesh S. Multi-objective Bayesian optimization with preferences over objectives. Advances in neural information processing systems. 2019;32.

[7] Taylor, Kendall, et al. "Bayesian preference learning for interactive multi-objective optimization." Proceedings of the Genetic and Evolutionary Computation Conference. 2021.

[8] Lin, Zhiyuan Jerry, et al. "Preference Exploration for Efficient Bayesian Optimization with Multiple Outcomes." International Conference on Artificial Intelligence and Statistics. PMLR, 2022.


clarity,_quality,_novelty_and_reproducibility: The novelty is limited since the paper uses a combination of previous approaches. 

Reproducibility is also concerning since the paper reports only three runs. 
summary_of_the_review: The paper addresses an important problem however, the novelty and experimental setup are limited.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Practical molecule generation involve optimization of multiple objectives simultaneously. These objectives are often expensive to evaluate making sample-efficiency key. The paper proposes a multi-objective Bayesian optimization method leveraging GFlowNets for optimizing the acquisition function. GFlowNets learn stochastic policies to generate discrete objects proportionally to their rewards, resulting in diverse candidates. The authors consider a preference-based decomposition of the MOO problem. The paper proposes a hypernetwork-based parameterization for conditioning on preferences. The authors also propose a hindsight-experience replay based strategy for leveraging offline data during learning. The authors use an evidential regressor as the surrogate model within the multi-objective Bayesian optimization context. The authors then present results on a molecule generation task with 4 objectives. 
strength_and_weaknesses: **Strengths**

- The paper tackles an important and challenging problem of multi-objective optimization in the context of molecule generation. As shown in previous work, using GFlowNets for optimizing the acquisition function results in diverse candidates and sample-efficient optimization
- The hypernetwork-based approach is an interesting way to implement conditioning, in contrast to FiLM based approaches. 

**Weaknesses**
- The paper overall is not very clearly written (I discuss this in more detail in the next section)
- The paper uses the preference-conditional GFlowNet formulation originally proposed in [1], but does not cite the paper where the preference-conditional GFlowNet is introduced. There are also other other inconsistent citations which I discuss in the next section,
- Multi-Objective REINFORCE [2] is very closely related to the proposed HN-GFN approach, differing only in the learning objective, but is not discussed / included as a baseline. Additionally the authors also do not include recent approaches such as LaMOO [3] in the baselines. 
- Aside from the baselines, the experiments seem somewhat limited. While the method enjoys superior performance in the task studied in the paper, it is not clear how well it generalizes to different settings and even different rewards for instance. The authors also provide only limited ablations to investigate the method. For instance it is not clear how the set of preference vectors is selected and what is the effect of the distribution of preference vectors used in training. 
- Minor: The authors use evidential regression for the surrogate model claiming "evidential deep learning presents the
advantages of faster inference speed and superior calibrated uncertainty" however, recent work [4] has established that such approaches can be arbitrarily miscalibrated. 


[1]  - GFlowNet Foundations

[2] - Pareto Set Learning for Neural Multi-Objective Combinatorial Optimization

[3] - Multi-objective Optimization by Learning Space Partitions

[4] - Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation
clarity,_quality,_novelty_and_reproducibility: **Clarity**

For the most part the discussion in the paper is clear. However, there are several places where prior work is incorrectly cited / missed out completely. In addition to the preference-conditional GFlowNet mentioned in the previous section, on line 4 on page 4, the authors cite Daulton et al. 2020 for the basic terminology of MOO instead of classic work in MOO [1,2]. On the second line in the paragraph before equation 3 the authors cite Daulton et al 2020 again for  "To support parallel evaluations in BO, one can obtain candidates according to different scalarizations", however, Daulton et al. 2020 does not consider scalarization at all. In fact it is other work [3] which establishes such approaches. Aside from this, there are aspects of the method which are also not clear. For example, the authors mention they use UCB as the acquisition function, but do not discuss how the UCB is used in this multi-objective setting - is the UCB applied to each objective individually or to the scalarization?

**Quality and Novelty** 

While some of the underlying ideas are not particularly novel - preference conditioning was introduced in [4] and GFlowNets in the context of BO was studied in [5] - the hypernetwork-based conditioning and hindsight experience replay are novel in the context of GFlowNets. However, as discussed in the previous section the empirical evidence is not substantial enough. 

**Reproducibility**

The authors do not provide code with the submission. The appendix does contain some relevant hyperparameters but some implementation details are not discussed at all. For example, no details are discussed about the training of the surrogate model - for instance whether the surrogate for each property is trained independently or a single multi-task model is trained, as well as other training details. The authors also do not mention other important hyperparameters like the UCB parameter (controlling exploration and exploitation). 


[1] Multicriteria optimization, Ehrgott, 2005

[2] Nonlinear Multiobjective Optimization, Miettinen, 2012

[3] A Flexible Framework for Multi-Objective Bayesian Optimization using Random Scalarizations

[4] GFlowNet Foundations

[5] Biological Sequence Design with GFlowNets
summary_of_the_review: In summary, while the paper presents an interesting GFlowNet-based approach to tackle multi-objective optimization, there are several major shortcomings in the paper in terms of the empirical analysis, baselines and framing of contributions. In the current state I lean towards rejection but encourage the authors to incorporate the feedback to improve the paper during the discussion.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method for multi-objective Bayesian optimization for molecular optimization. The proposed method uses GFlowNets to optimize the acquisition function in BO, and uses a hypernetwork-based method to incorporate the preference vector into GFlowNets such that a diverse set of points can be sampled from the Pareto front.
strength_and_weaknesses: Strengths:
- The proposed method is intuitive and modifies GFlowNets in a reasonable way to facilitate multi-objective optimization.
- The experiments are nicely done, and the experimental results look promising.
- The paper is in general well written, and the proposed method is well motivated.

Weaknesses:
- I feel that some of the algorithmic details are not clearly explained, particularly the connection between Algorithm 1 (for training HN-GFN) and BO. For example, does the dataset $\mathcal{D}$ correspond to the currently available observations from all previous iterations of BO? Does the reward function $R$ here correspond to the acquisition function calculated in BO? How is the set of target preference vectors built? More importantly, do you need to run Algorithm 1 after every iteration (or every batch) of BO? If yes, then the computational costs may become an issue and hence should discussed.
- I think Section 4.3 needs to be revised to make it clearer, in the current form, it's not easy to understand.
- Top paragraph of page 2: it's still unclear to me why limitation 1) makes "existing discrete molecular optimization methods" not applicable as "acquisition function optimizer". Can't you simply use those acquisition functions which directly take diversity into account? For example, if you use the GP-BUCB acquisition function from paper [1] below, to select an input in a batch, you can simply invoke an existing discrete molecular optimization method to maximize the acquisition function (whose GP posterior standard deviation is updated every time a new input is selected), which will naturally lead to a diverse set of inputs in a batch.       
[1] Parallelizing Exploration-Exploitation Tradeoffs in Gaussian Process Bandit Optimization, JMLR 2014
- Top paragraph of page 9: the number of rounds $N=8$ is in fact unusually small in BO, and the batch size $b=100$ is also unusually large for BO as well. Are these choices the common practice in molecular optimization using GFlowNet?
- (minor) In the Related Work section, the previous works on multi-objective BO should also be discussed.

clarity,_quality,_novelty_and_reproducibility: Clarify: The paper is well written in general, but some of the algorithmic details can be better explained, as I discussed above.

Quality: The algorithmic design and the experiments are of high quality.

Novelty: The problem of multi-objective BO for molecular optimization is intuitive and hence not novel, but the use of the hypernetwork to condition on different preference vectors in GFlowNets is novel as far as I know.

Reproducibility: Some experimental details are discussed, but the code is not uploaded.
summary_of_the_review: The paper solves an important problem for molecular optimization using GFlowNets, and I don't have major concerns about the paper. The concerns I listed under "Weaknesses" are mostly regarding the writing of the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work investigate the possibility of using GFlowNets to tackle multi-objective sampling problems. Specifically, hypernetwork-based GFlowNet is proposed for solving multi-objective Bayesian optimization problems. Some insights from reinforcement learning is also involved. The proposed algorithm is evaluated in molecule generation tasks.
strength_and_weaknesses: ## Strengths
1. The methodology is well-formulated, with careful design for amortizing multi-objective problems and surrogate modeling.
2. The off-policy property of GFlowNet is examined, including a mix of different training trajectory distribution, and the use of hindsight replay idea.

## Weaknesses
1. I would expect some illustration / analysis on at least toy tasks to show that the proposed method indeed learn the multi-objective behavior. E.g., when the conditioning preference is changed, the distribution of GFlowNet will change accordingly.
2. The diversity should be measured for every fixed preference. If allowed different preference, there will be of course diverse samples generated by the GFlowNet. 
3. Is there any particular reason to model the preference conditioning with a hypernetwork?  Hypernetwork is not very stable in training. What's more, usually it is enough to directly augment the input with preference for the conditioning, which is also much easier. I would expect the authors to justify this choice with empirical evidence.
4. Is there any particular reason to use evidental regression? According to my experience, sometimes it is not as reliable as more "traditional" methods like deep ensemble. Also, it seems HierVAE-based methods and GP-BO still uses GP, while the proposed method uses evidental regression. Is this an unfair comparison? I would expect at least some results about using HN-GFN with GP to justify this choice.

### Minors
- The experiments are limited to molecule generation. The multi-objective problems are much more than small graph generation. Other applications, such as protein, sequences, are also of great importance. 
- Is the set of target preference vectors $\Lambda$ kept fixed? If not, how to update it?
- In Fig.2 (left), why HN-GFN could be better than the "gold-standard" preference-specific GFlowNet? Do these two use the same GFlowNet architecture?
- Fig.2 (right) shows the performance with $\gamma=0, 0.2$, but it would be great to see a spectrum from 0 to 1. This is important for the choice of hyper parameter $\gamma$, which seems not mentioned in the paper. Please correct me if I miss anything!

clarity,_quality,_novelty_and_reproducibility: ## Clarity & Quality
Most parts of the paper are clearly written. Some details are missing, though, for example $\gamma$.

## Novelty
The idea is actually not entirely, but has been long mentioned as "Pareto GFlowNet" in [GFlowNet Foundations](https://arxiv.org/pdf/2111.09266.pdf).

## Reproducibility
I cannot check the reproducibility as it seems the implementation is not provided.
summary_of_the_review: The method of this paper is clean and straightforward, while this work still suffers from non-extensive experiments and insufficient ablation to justify some usage of the components. I understand that it is a tight schedule for the authors to rebuttal, but I would of course consider raising the score if some of the main concerns are addressed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper considers the problem of molecule design to optimize multiple objectives with an emphasis on evaluating a batch of molecules in each round. The solution employs the hypernetwork-based GFlowNets as acquisition function optimizer and scalarization based acquisition function. This is a very important problem.

There is a lot of work on Bayesian optimization over 1) combinatorial spaces; and 2) multi-objective optimization. So a natural approach would be to combine the best methods from #1 and #2 in a synergistic manner. Some candidates are combining state-of-the-art latent space BO methods (e.g., LOL-BO, LADDER) from #1 and "Diversity-guided multi-objective bayesian optimization with batch evaluations." Advances in Neural Information Processing Systems 33 (2020): 17708-17720 from #2. Similarly, GP based surrogate models using fingerprint kernel with USeMO from #2. The paper has some candidate baselines, but they may not be the best ones available in this spirit and/or employed appropriately (e.g., qEHVI works very well in practice)

In this problem setting, we want to see the sample-efficiency of the approach. The typical evaluation procedure to show performance curves with hypervolume or diversity on y-axis and number of evaluated molecules on x-axis. In the past, researchers have shown that RL based approaches are not sample efficient for optimization problems with expensive evaluations. These performance curves are missing in the paper. 

The novelty of the technical approach is somewhat limited as it heavily relies on leveraging prior methods. The related work on BO over combinatorial spaces and multi-objective BO does not reflect the current state of knowledge -- lot of missing references (some of them are pointed out by the reviewers). Please see the reference list from the recent NeurIPS-2022 tutorial on Bayesian Optimization. Additionally, there should be some qualitative discussion of why the proposed method should perform better than a Bayesian optimization solution, which is missing.

The paper is promising, but it falls short of acceptance for the above reasons. Hence, I recommend rejecting the paper. I strongly encourage the authors' to improve the paper based on the review feedback for resubmission.
Decision: Reject