Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: A recent 'bidirectional learning' approach was proposed for offline model-based optimization. This paper extends it with a few tweaks: instead of using an NTK kernel, use the linearization of a pretrained foundation model, and tune optimizer hyper-parameters automatically by gradient descent using an auxiliary model to provide pseudo-labels for held-out data. The experiments focus on protein/dna design tasks and consider a number of methods from recent work.
strength_and_weaknesses: =Strengths=
Paper contributes to an existing active thread on offline model-based optimization, which is an interesting and important area.
Paper tackles offline biological sequence design, which is a practical application with large potential impact.
Paper combines a number of modern techniques, such as neural network linearization, gradient-based hyper-parameter tuning using bilevel optimization, etc.

=Weakness=
I found the exposition of the bidirectional learning method extremely confusing.
See my full review below regarding my hesitations around the setup of the experiments.
clarity,_quality,_novelty_and_reproducibility: The paper combines a couple of orthogonal ideas to extend a recent paper on bidirectional model-based optimization (Chen 2022). Each of these ideas is clever, but has appeared in other ML contexts. 

I found the exposition of the bidirectional learning method very confusing. It can only be understood by reading the original paper.

I have some large reservations regarding the quality of the analysis of the experimental results (see below).



summary_of_the_review: A have some important reservations about the paper below.
 
==Parametrization of the oracle vs. the regression model==
For datasets where the entire search space hasn't been measured (i.e., everything other than the tfbind datasets), you define the ground truth f(x) by fitting a regression model (the 'oracle') on the entire dataset, such that this model can be queried on any x. I'm very concerned that the same overall model architecture is used for the oracle, the methods' function approximators f_theta, and f_aux: 
*Oracle: "The oracle passes the average of the residue embeddings from the pre-trained Prot-T5 (Elnaggar et al., 2021) into a linear layer and then fits the dataset. The following two task oracles take the same form."
*Function approximator: "In this paper, we adopt the pre-trained DNABERT (Ji et al., 2021) and Prot-BERT (Elnaggar et al., 2021) models, and compute the average of token hidden embeddings as the extracted feature, which is fed into the linear layer to build the proxy. … Following (Dukler et al., 2022), we can also only linearize the last layer of the network for simplicity."
*Auxiliary model f_aux: ""We implement the auxiliary model as a linear layer with the feature from the pre-trained LM?"

This setup will unfairly favor methods that don't rely on a function approximator (i.e., it will penalize methods like CMA-ES), since the function approximator will have low approximation error for fitting the oracle, since they're from the same function class. It will additionally favor methods that use f_aux and model linearization, since these closely mirror the structure of the oracle. This is not a realistic setup. Model-misspecification will always be present when trying to model real experimental data with a neural network.

An alternative approach would avoid the oracle by evaluating optimization methods' ability to rank a held-out set of examples from the dataset (ranking them by the methods' associated acquisition function). I would have preferred to see some experiments seeing the second approach, since it removes the above issues.

==Presentation of the paper's method==
I found section 3.1 extremely confusing. There should be a background section that explains the bidirectional learning approach fully. The paper should be self-contained. Right now, readers will not be able to understand the explanation of how it works. The datasets Xh and Xl are never defined. I was completely unclear to me where yh comes from. I assumed that Xh were prospective designs and yh was unobserved. It was also never explained how the two models are combined into an acquisition function.

==Complex model parameterization==
I work in this exact research field. In my experience, simple linear models are often quite competitive when modeling these datasets. I would have appreciated some demonstration that the particular modeling choice you employ is sensible.

==Presentation of experimental results==
It was very difficult for me to understand if any of the difference between methods' performances in Table 1 are statistically significant. As far as I can tell, all of these optimization methods are randomized. How much variance in performance is due to this randomness? I would have appreciated some error bars that convey this variance.

==Role/Importance of f_aux==
sec 3.2 is something you could do for basically ML system: just train two models and use M1 to provide aux labels for hyper tuning. for this to work, the aux model needs to be accurate, but if you're able to fit a good aux model, then isn't the overall problem easy?

==Additional issues==

"Both DNA tasks have exact oracles for ground-truth evaluation."
This isn't quite true. The datasets measure the entire search space, but these measurements come from a noisy wet-lab experiment. We never observe the true noiseless value.

Fig 2 was not compelling to me. Where in sec 3.2 does gamma depend on T? Are you simply plotting the trajectory of the learned scalar gamma over the course of hyper-tuning. The shape of this trajectory should not be important.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new method for offline model-based design, which is then applied to biological sequences. The method is able to use a kernel based on pretrained LM features, which allows the method to leverage additional knowledge. In addition, a learning rate adaptation method is proposed for bidirectional learning, with results showing improvement on the chosen benchmarks.
strength_and_weaknesses: Strengths
* New gradient based optimization scheme that can leverage pretrained language models
* Interesting approach for adapting the learning rate, which helps to deal with varying protein landscapes
* Math is correct and is fairly easy to follow
* Results appear relatively strong

Weaknesses
* Something is odd with the scale of the experiments. When I download the avGFP dataset, the maximum fitness value in the dataset is ~4.12, while the paper reports a value >8. The 4.12 number also appears consistent with the results in Ren et al. 2022. There also seems to be a scale mismatch in the AAV and E4B datasets between this paper and Ren et al. 2022 (in this paper, AAV fitness varies between 0.4 and 0.6, in Ren et al., it varies between -3.5 and 4.5; something similar is true for E4B).
    - It would be useful if the authors listed the exact fitness metric for each task and perhaps the maximum value achievable in the dataset. Additionally authors should explain or correct any scale mismatch when comparing with previous work.
    - This is the largest issue that must be addressed before publication - I would increase my score if this is addressed satisfactorily.
* The authors argue that the incorporation of pretrained language model features is a point in favor of their method. It would be interesting to see how the method performs as a function of language model quality. You could use different models from the Elnaggar et al. or the various sizes of the ESM language models and see if improvements in language modeling result in improvements in optimization, which would more directly show that these features have a significant effect on performance.
* Hard to follow what metrics are being reported in the table - could this be added to the table caption? It’s actually never explicitly said that the reported number is maximum fitness achieved, but I’m assuming it is
* It would be nice to motivate the problem setting better. I have a hard time understanding when this class of method would be useful. It purports to very quickly optimally sample a set of designs (# iterations is set to 25 in this evaluation). But it also requires a dataset of >1000 training points to start out (even if this is limited to low scoring points). That suggests that you 1) must already have a high throughput way of screening fitness values for this task but also 2) want to further optimize a very constrained number of samples. In the case where you are able to generate the low-scoring dataset of size 1000+, does it matter if you find the optimal solution after 25 iterations or after 500 iterations?
* Related to the above point, it would be good to see the performance of the method as a function of the size of the initial training dataset. How well does the method function in a true low-N setting (e.g. < 100 total samples).

Minor Comments
* Section 4.4, points (4) and (5) argue that the length of the sequence is a factor in whether gradient-based methods have a large advantage  over other methods. While I agree this makes sense intuitively, I'm not sure the evidence in the paper supports it. Gradient-based methods still seem to outperform on the DNA tasks, which have a much smaller search space. Additionally, the gap is larger in TFBind10 vs. TFBind8, but not clear exactly how much larger or whether this is actually significant.


clarity,_quality,_novelty_and_reproducibility: Clarity: The method in the paper is clearly described, and I could follow the math. The experiments are not as clearly described. Some details are missing, and it's not clear what fitness metrics are being optimized.

Quality: The method is interesting and theoretically sound. It's difficult for me to judge the quality of the results due to the scale discrepancy described.

Novelty: I believe this method is novel, but I don't actively work in optimization and may not be aware of related work.

Reproducibility: Code is provided for the model. Reproducibility issues exist due to the scale discrepancy described.
summary_of_the_review: The method is well presented and novel, and would be a reasonable contribution to ICLR. There are some issues with the evaluation and the numbers in the table appear not to be correct (or they are not correctly explained in the paper). If this issue is fixed, the paper would be a good candidate for acceptance, but it must be fixed first.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper tackles the model-based optimization problem for biological sequences. They extend the NTK-based bidirectional learning approach from previous work which essentially attempts to regularize according to features that transfer between low-fitness and high-fitness sequences. Their main contribution is twofold (1) they extend Chen et al 2022 by replacing the NTK representation with a pretrained language model with a linear layer atop and (2) they propose adaptive hyperparameter tuning strategies to improve the optimization scheme. They then benchmark their method against biological sequence datasets.
strength_and_weaknesses: Strengths
- The paper puts together an interesting set of ideas(bilevel optimization, deep linearization) that help resolve a weakness of prior work.
- The paper seems to perform better on benchmark for biological sequence design tasks.
- Adaptive hyperparameter tuning is extremely useful in model-based optimization settings due to the difficulty of tuning hyperparameters on out-of-distribution datasets.

Major Weaknesses
- Lack of clarity on major experimental details makes interpretation of results difficult: Is using the pretrained LM for all proxy methods shown to perform better than a standard CNN-type model as this has been shown to work well in FLIP (Dallago et al 2021)? Is there finetuning done on the LM used as a proxy for other methods? What is a "task oracle"? How were X_h and y_h initialized?  
- The central premise of the paper is that the following rank order is expected for surrogate model models: a NN(perhaps finetuned version of pretrained LM ) which is intractable to optimize against > linearized pretrained LM > NTK. However, this is never shown in any ablation studies rather intuition about biophysical features are referenced. The paper would be strengthened a lot if it was shown that such a representation indicates a better surrogate model by demonstrating improved prediction performance(rather than optimization) on the biological sequence datasets shown later. This is because there is significant stochasticity in the ranking of the MBO methods depending on the experimental setting and hyperparameter choices so a better understanding of the central claim of the paper would


Minor Weaknesses:
- Confidence intervals should be provided for the scores.
- How were indels handled in the AAV task (given the biological sequence representation assumes fixed length $L$)?
clarity,_quality,_novelty_and_reproducibility: The paper is largely experimental in nature by taking a previously published framework (Chen et al 2022) and applying a number of bells and whistles (bilevel optimization with adaptive hparam tuning, deep linearization, etc) to get better performance. However, lack of clarity in the exposition and training details makes the.
- The exposition would be improved with a diagram of the architecture setup for the Deep Linearization (and a general workflow diagram) as is common in DL papers.
- More care could be taken with the presentation of the equations to make the paper more readable.

summary_of_the_review: The paper has a nice set of ideas that could plausibly provide an improvement upon bidirectional learning proposed in (Chen et al 2022). However, the lack of care taken toward the presentation and experimental section in particular significantly hampers the ability of the paper to be published in its current form.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers and meta reviewer all carefully checked and discussed the rebuttal. They thank the authors for their response and their efforts during the rebuttal phase.
The response helped resolve some concerns (e.g., further experimental details and some analysis of the ranking order of the surrogate models).

The reviewers and meta reviewer all acknowledge that the submission tackles an important problem, namely the (model-based) optimization of biological sequences.

This being said, the submission still suffers from important flaws that would warrant further consolidations/investigations. In particular:

(i) The contributions 2 and 3 (“adaptive-gamma” and “adaptive-eta”) do not seem to relate at all to the vast existing literature about bi-level optimization addressing similar problems, e.g., (Maclaurin et al., 2015; Luketina et al., 2016; Pedregosa, 2016; Franceschi et al., 2018; Lorraine et al., 2020) to name a few.  As a result, it makes it difficult to assess the substance of those two contributions. For example, “... the inner level solution can be approximated via a gradient descent step with a learning rate…” seems to correspond to the method of Luketina et al., 2016. Similarly, the extension to the optimization of the learning rate (“adaptive-eta”) has strong connections with work like Donini et al., 2020 (MARTHE: Scheduling the Learning Rate Via Online Hypergradients) & references therein.
To make contributions 2 and 3 actual contributions, precise discussions of the related work and extensive comparisons should be carried out (especially because the claims in the submission are strong, e.g., “...the first learning rate adaptation module for offline model-based optimization…”).

(ii) Regarding contribution 1, it would be important to more clearly state what is inherited from Chen et al. 2022 and what is novel in the submission. Several sections appear to be heavily inspired by Chen et al. 2022 (e.g., the ablation study about forward vs. backward).

(iii) The exposition of the methodology needs to be clarified and to be made more formal (along the lines of the exposition in Chen et al. 2022). Also some expressions seem to be mistaken (e.g., lack of second-order derivative in equation (21)). Analogously, the equation (6) introduces non-differentiability (argmax), whose implications are not discussed.

Because of the extremely competitive landscape of the submissions this year, the paper remains under the cut and has ultimately not been selected for acceptance.
We are convinced that the suggestions above will help strengthen the paper for a future resubmission, which the reviewers and meta reviewer all encourage.

As a final suggestion, with the paper being centered around protein design, there are differences between optimizing a synthetic neural-network landscape and a true protein fitness landscape. In the current state, we have little evidence that the methods being currently developed will succeed on both the former and the latter. Since the former is an arguably very difficult goal, it would be better to frame/tone down the narrative of the paper and set expectations accordingly, e.g., saying that the paper primarily focuses on a proxy, synthetic problem.  

justification_for_why_not_higher_score: * Contributions not well supported.
* Differences with respect to closely-related work not sufficient to properly assess the contributions.
* Clarity/formal presentations.
Decision: Reject