Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper investigates the subject of Offline Model-Based Optimization, which involves processing a static dataset of designs X and function evaluations Y in order to solve the optimization problem X* = \argmax_{X} f(X) for an unknown function f. This paper proposes a gradient-based method for solving Offline MBO problems using infinite-width DNN models. The approach facilitates efficiently optimizing designs X via a proposed bidirectional objective that encourages designs to be found that not only achieve high performance under an approximation of the objective function f_{\theta} (X), which is referred to as the forward mapping in the paper, but jointly maximize a distillation objective that encourages optimized design-score pairs to be informative about the characteristics of the true objective function (the backward mapping).

One key novelty in the paper is the proposed use of a distillation objective to constrain the optimized design-score pairs, rather than regularizing the DNN model used for optimization. Results in the paper appear to show the effectiveness of this principle on a standard benchmarking framework for Offline MBO using a comprehensive set of baselines.
strengths_and_weaknesses: Originality:

The subject of mitigating out-of-distribution samples (ie, design-score pairs) using gradient-based Offline Model-Based Optimization techniques has been the subject of multiple papers. The subject itself is not novel, but the solution proposed by the authors (regularizing the optimized samples rather than the model) is an original proposition in the Offline MBO domain. Though different in execution, a similarly inspired technique has been proposed in the Offline RL setting, namely BCQ (Fujimoto, S. et al., 2019), which constrains the (s’, a’) pairs sampled during bellman backups \max_{a’} Q(s’, a’) to be contained in the support of the static dataset.

Both papers explore constraints (either implicitly or explicitly) on the data being optimized (actions in the case of BCQ, and designs in the case of BDI), rather than model regularization. However, it should be noted that the implementation of the constraint is fundamentally different between these two approaches, and BDI’s implementation is original. Dataset distillation enforces a constraint on the data that is an appealing alternative to the support-based constraint seen in BCQ because it lets the model generate designs outside the training distribution.


Quality:


The experimental setup in the paper adheres to the evaluation protocol established in prior benchmarking papers, which aids in the reproducibility of the paper. Relevant baselines from the literature, including several new methods, and multiple classical methods, adapted to the Offline MBO setting, are reported. These factors improve the quality of the paper’s main evaluation. Certain relevant ablations are not included (see my questions below), and a small number of claims and speculations made in the paper are not fully supported by evidence. For example, on line 278-279 of the manuscript, the authors write “we observe that the backward mapping is more effective for continuous tasks, possibly because we do not model the sequential nature of DNA and protein sequences.” This is a helpful intuition, and could be improved if the authors tested this hypothesis with a model that does “model the sequential nature of DNA and protein sequences.” Quality may also be improved by addressing the questions listed below.

Clarity:


Certain experimental details are not clear from the manuscript. For example, on line 194-195 the authors write “we choose the top N = 128 most promising designs for each method,” which could suggest more than N = 128 designs are generated, and the number is downsampled to 128 using some indicator for how promising the designs are. If such a post-selection mechanism is used, additional details should be included in the paper for reproducibility. Such a mechanism should query the ground truth objective function only N = 128 times to compute 100th percentile and 50th percentile statistics as per design-bench (Trabucco, B. et al., 2022).

Significance:


Mitigating the out-of-distribution problem in Offline MBO has significance as a research area, due to its compelling applications in real-world design problems. The results in this paper show the proposed approach is quite effective in practice at finding high-performing designs.
questions: (1) There appears to be a discrepancy between a small fraction of the results presented in the main performance table, and those reported in other recent papers. In particular, the performance of MIN and COMs are different from those reported in design-bench (Trabucco, B. et al., 2022) on the following tasks: Ant Morphology, DKitty Morphology, and Hopper Controller. Could the authors investigate this and explain the cause of this difference?

(2) The optimization objective in Equation (4) is chosen as a regression objective, rather than purely maximizing the predictions of the learned objective function f_{\theta} (X). How are the results affected when this equation is changed to pure maximization, rather than regression?

(3) The objective functions in Equation (8) appear to be assigned equal weight. How are the results affected by ablating the weight of the distillation term (Equation 6) in the loss function?
limitations: The authors have sufficiently addressed these.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Offline model-based optimization could suffer from the distributional shift issue where the model bias could be exploited by the optimization process to output poor designs. Most previous offline model-based optimization aims to solve the problem by introducing loss function terms that promote distribution matching between the learned model outputs and the ground-truth. Unlike previous works, this paper introduces a directional learning approach based on neural tangent kernel to facilitate the distribution matching. By encouraging learning accurate mappings from low-scoring designs to high-scoring designs and vice versa, the proposed method could alleviate the issue of outputting unseen designs with significantly overestimated scores. The proposed method achieved the best performance on six out of seven widely studied offline model-based optimization task.
strengths_and_weaknesses: ### Strengths
1. The proposed approach is well motivated and the writing is easy to follow.
2. The authors provide a good insight on why the bidirectional learning helps mitigate the distributional shift issue in 3.3.
3. Experiment results show strong performance of the proposed method.

### Weakness
This paper lacks a discussion on the relationship between the directional learning and a conservative regularization. I believe that adding the backward mapping loss is similar to a conservative regularizer in the sense that they both aim to alleviate the distributional shift between the predicted optimal designs and the dataset designs. I hope that the authors could provide some discussions on their similarity/difference and why learning the backward mapping could be a better alternative than conservative regularization methods.

questions: Could the authors explain why using a constant $y_h$ could work? Would it be possible to further improve the proposed method with a more sophisticated design of $y_h$ where the values are different for each high-scoring sample?
limitations: The authors have addressed the limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes bidirectional learning for offline infinite-width model-based optimization (BDI) to address the out-of-distribution problem. BDI consists of a forward mapping and a backward mapping. The authors claim that the backward mapping can distill more information into high-scoring designs. Experiments show that BDI achieves state-of-the-art performance in both continuous and discrete tasks.
strengths_and_weaknesses: Originality:

The idea of BDI is novel. This paper introduces the idea of neural tangent kernels into offline model-based optimization. Moreover, the authors propose a novel backward mapping, significantly improving the performance of model-based optimization.

Quality:

The comparisons are thorough. Results show that BDI outperforms baselines in most tasks, and all component in BDI is useful. The use of neural tangent kernels in BDI is based on solid theoretical motivation. Nevertheless, I am confused about the backward mapping.
1. What is the motivation for using the backward mapping? The authors may want to provide an intuitive explanation of why we need a backward mapping besides the forward mapping.
2. How does the backward mapping help to distill information into the high-scoring designs? Though the authors provide ablation studies to show that the backward mapping does work, the authors may want to provide more analyses/visualization to show why the backward mapping works.
3. I can not understand the predefined score $y_h$ used in the backward mapping. In forward mapping, a high $y_h$ can encourage the algorithm to generate $X_h$ with high scores. In backward mapping, what is the purpose of training $f_\theta^h$ with a virtual score $y_h$?



Clarity:

Most of this paper is easy to follow. Nevertheless, I am confused about the backward mapping. The authors may want to discuss the backward mapping more, especially its motivation and how it works.

Significance:

The backward mapping is novel and can provide a significant performance improvement. 
questions: Please refer to the "Strengths And Weaknesses" part.
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: This paper studies Offline Model-Based Optimization. This paper proposes a gradient-based method for solving Offline MBO problems using infinite-width Deep learning models.
The key novelty of the paper is in proposed use of a distillation objective to constrain the optimized design-score pairs.

All three reviewers identify the novelty of the problem and the approach. 
The paper also presents strong empirical evaluation on standard benchmarks. 

The rebuttal discusses yielded constructive changes in the paper, and the authors are expected to account for the discussion and suggestions in the next iteration of the manuscipt. 
The AC concurs with the reviews and the discussion thereafter. 
confidence: Certain
Decision: Accept