Parallel-mentoring for Offline Model-based Optimization
Authors: Can Chen
Christopher Beckham
Zixuan Liu
Xue Liu
Christopher Pal
Venue: NeurIPS 2023 poster (2023)
Keywords: offline model-based optimization
bi-level optimization
TL;DR: We propose a simple and novel method called parallel-mentoring to improve the ensemble robustness by facilitating the exchange of weak ranking supervision signals among proxies.

Abstract: We study offline model-based optimization to maximize a black-box objective function with a static dataset of designs and scores. These designs encompass a variety of domains, including materials, robots, DNA sequences, and proteins. A common approach trains a proxy on the static dataset and performs gradient ascent to obtain new designs. However, this often results in poor designs due to the proxy inaccuracies for out-of-distribution designs. Recent studies indicate that (a) gradient ascent with a mean ensemble of proxies generally outperforms simple gradient ascent, and (b) a trained proxy provides weak ranking supervision signals for design selection. Motivated by (a) and (b), we propose $\textit{parallel-mentoring}$ as an effective and novel method that facilitates mentoring among proxies, creating a more robust ensemble to mitigate the out-of-distribution issue. We focus on the three-proxy case in the main paper and our method consists of two modules. The first module, $\textit{voting-based pairwise supervision}$, operates on three parallel proxies and captures their ranking supervision signals as pairwise comparison labels. These labels are combined through majority voting to generate consensus labels, which incorporates ranking supervision signals from all proxies and enables mutual mentoring. Yet, label noise arises due to possible incorrect consensus. To alleviate this, we introduce an $\textit{adaptive soft-labeling}$ module with soft-labels initialized as consensus labels. Based on bi-level optimization, this module fine-tunes proxies in the inner level and learns more accurate labels in the outer level to adaptively mentor proxies, resulting in a more robust ensemble. Experiments validate the effectiveness of our method. Our code is available here.

Parallel-mentoring for Offline Model-based Optimization
Can (Sam) Chen Christopher Beckham Zixuan Liuf, Xue Liu Christopher Pal234
'cGGill University, "MILA Quebec Al Institute,
Polytechnique Montreal, *Canada CIFAR Chair, UUiiveritty of Washington
Abstract
We study offline model-based optimization to maximize a black-box objective function with a static dataset of designs and scores. These designs encompass a variety of domains, including materials, robots and DNA sequences. A common approach trains a proxy on the static dataset to approximate the black-box objective function and performs gradient ascent to obtain new designs. However, this often results in poor designs due to the proxy inaccuracies for out-of-distribution designs. Recent studies indicate that: (a) gradient ascent with mean ensemble of proxies generally outperforms simple gradient ascent, and (b) a trained proxy provides weak ranking supervision signals for design selection. Motivated by (a) and (b), we propose mentoring among parallel proxies, creating more robust ensemble mitigate the out-of-distribution issue. We focus on the three-proxy case and our method consists of two modules. The first module, voting-based pairwise supervision, operates on three parallel proxies and captures their ranking supervision signals as pairwise comparison labels. These labels are combined through majority voting to generate consensus labels, which incorporate ranking supervision signals from all proxies and enable mutual mentoring. However, label noise arises due possible incorrect consensus. To alleviate this, we introduce an adaptive soft-labeling module with soft-labels initialized as consensus labels. Based on bi-level optimization, this module fine-tunes proxies in the inner level and learns more accurate labels in the outer level to adaptively mentor proxies, resulting more robust ensemble. Experiments validate the effectiveness of our method. Our code is available here
as an effective and novel method that facilitates
Introduction
Designing new objects or entities to optimize specific properties a widespread challenge, encom- passing various domains such as materials, robots, and DNA sequences (I). Traditional approaches often involve interacting with a black-box function to propose new designs, but this can be expensive or even dangerous in some cases (2H6). In response, recent work (1 has focused on more realistic setting known as offline model- optimization (MBO). In this setting, the objective is to maximize black-box function using only a static (offline) dataset of designs and scores.
prevalent approach to addressing the problem to train a deep neural network (DNN) model parameterized as fe(-), the static dataset, with the trained DNN serving proxy. The proxy allows for gradient ascent on existing designs, generating improved designs by leveraging the gradient information provided by the DNN model. However, this method encounters issue with the trained proxy being susceptible to out-of-distributtin problems. Specifically, the proxy produces inaccurate predictions when applied to data points that deviate significantly from the training distribution.
"Conresonnencee to can.chen @ @aiea.quiiilucccccm
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Recent studies have observed that (a) employing a mean ensemble of trained proxies for gradient ascent in offline MBO generally leads superior designs compared to using a single proxy 7 This improvement stems from the ability of the ensemble
to provide more robust predictions compared to sin- gle proxy 8111 Recent work has also found that (b) a trained proxy offers weak (valuable, albeit potentially unreliable) ranking supervision signals for design selec- tion in various offline MBO contexts, such as evolutionary algorithms [22 reinforcement learning [3], and genera- tive modeling [14]. These signals, focusing on the relative order of designs over absolute scores, are more resilient to noise and inaccuracies. By exchanging these signals can potentially
fo > (e!)
(o())) 12(())
Majority Voding
(xf)) < (187))11(0)))
fg
Mentoring
Voted ranking signal f ("()) > (we))
among proxies in the ensemble, hance its robustness. As shown in Figure [ we have three parallel proxies (.). 1 (.) and fo (-). For two designs of and 22 within the neighborhood of the current optimization point, proxies fa)() and fø agree that the score of æf is larger than that of æ2 while proxy f-) disagrees. Based on the majority voting principle, proxies (-) and fø (-) provide more reliable ranking, and their voted ranking signal f (xf)= / (a:²) could mentor the proxy J (-), thus enhancing its performance.
Figure 1: Motivation illustration.
(.)
To this end, we propose an effective and novel method called parallel- parallel-mentoring that facilitates mentoring among parallel proxies to train a more robust ensemble against the out-of out-of-distribun of-distribution issue. This paper primarily focuses on the puree-proxy case, referred tri-mentoring. but also examine the situation with more proxies in Appendix I. As depicted in Figure 2 tri-mentoring consists of two modules. Module 1, voting-based pairwise supervision (shown in Figure 2 a)), operates on three parallel proxies fg((), (-), (-). and fo and utilizes their mean for the final prediction. To ensure consistency with the ranking information employed in design selection, this module adopts a pairwise approach to represent the ranking signals of each proxy. Specifically, illustrated in Figure a this module generates samples (e.g. aT. az and az) in the neighborhood of the current point a and computes pairwise comparison labels ÿd for all sample pairs, serving as ranking supervision signals for the proxy (-). The label ø is defined as if fo (x,) > fo (x;) and O otherwise, and similar signals are derived for proxies fo (.) and f.). These labels ÿ G and ÿ are combined via majority voting to create consensus labels ÿ which are more reliable and thus can be used for mentoring proxies. The voted ranking signal fV (x+') > f (xx)) Figure corresponds to the pairwise consensus label D in Figure 2 a). and both can mentor the proxy Sg (.).
Neighborhood sampling
Parallel praxies
8
Praxy
f
Fine- praxy
(d())) f805)
Soft label update
Offliee supervision
Consensus label
81
0%
Inner: fine-tuning
6/2
Majority Voting
Static dataset
f
8
f
O
f
Soft label s
0.9
Outer: soft-labeling
Pairwise comparison labels
03
Mentoring
au
6.7
(a) Vating-based Pairwise Supervision
(b) Adaptive Soft-labeling
Figure 2: Illustration of tri-mentoring.
Module 2, adaptive soft-labeling (shown in Figure 2b)). mitigates the issue of label noise that may arise, since the voting consensus may not always be correct. To this end, this module initializes the consensus labels ÿ" from the first module as soft-labels ÿ". It then aims to learn more accurate soft-labels to better represent the ranking supervision signals by leveraging the knowledge from the static dataset. Specifically, assuming accurate soft-labels, of the proxies, either fo (-). f or fo (-), fine-tuned using them, is expected to perform well the static dataset, as both soft-labels

---

(pairwise perspective) and the static dataset (pointwise perspective) describe the same ground-truth and share underlying similarities. This formulation leads to bi-level optimization framework with an inner fine-tuning level and an outer soft-labeling level as shown in Figure 2b). The inner level fine-tunes the proxy with soft-labels, which establishes the connection between them. The outer level optimizes soft-labels to be more accurate by minimizing the loss of the static dataset via the inner-level connection. The optimized labels are further fed back to the first module to adaptively mentor the proxy, ultimately yielding a more robust ensemble. Experiments design-bench validate the effectiveness of our method.
To summarize, our contributions are three-fold:
We propose paraolel-mentoringg for offline MBO, effectively utilizing weak ranking supervision signals among proxies, with a particular focus on the three-proxy case as tri-mentoring
Our method consists of two modules: voting-based pairwise supervision and adaptive soft- soft-labeling The first module generates pairwise consensus labels via majority voting to mentor the proxies.
To mitigate label noise in consensus labels, the second module proposes bi-level optimization framework to adaptively fine-tune proxies and soft-labels, resulting in more robust ensemble.
Preliminaries: Gradient Ascent on Offline Model-based Optimization
Offline model-based optimization (MBO) aims to find the optimal design xx that maximizes the black-box objective function f(-):
arg max f(x),
To achieve this, static dataset D {(x, y:) with N points is available, where æ; represents design and yi is its corresponding score.
A common approach for solving this optimization problem is to fit a deep neural network (DNN) model (-(-) with parameters to the static dataset supervised manner. The optimal parameters O" can be obtained by minimizing the mean squared error between the predictions and the true scores:
r- = arg min N
(folz.)) - y1)2.
The trained DNN model fo- (-) acts proxy to optimize the design using gradient ascent steps: Ft+1 Dr + 1 fe(() 2= for t [0, - 1],
where is the number of steps and r represents the learning rate. or serves as the final design candidate. However, this method faces challenge with the proxy being vulnerable to out-of- distribution designs. When handling designs that substantially differ from the training distribution, the proxy yields inaccurate predictions.
3 Method
In this section, we introduce parallel-mentoring, focusing on the three-proxy scenario, also known as tri-mentoring. The method can be easily extended to incorporate more proxies, as discussed in Appendix A.I Tri-mentoring consists of two modules. The first module, voting-based pairwise supervision in Section 3.1 manages three proxies parallelly and generates consensus labels via majority voting to mentor proxies. To mitigate label noise, we introduce a second module, adaptive soft-labeling in Section 3.2 This module adaptively fine-tunes proxies and soft-labels using bi-level optimization, improving ensemble robustness. The overall algorithm shown in Algorithm 1
3.1 Yoting-based Pairwise Supervision
We train three parallel proxies fgl (:), fo (-) and (-) on the static dataset with different initializations and utilize their mean as the final prediction as suggested in [
1 fe - (
fo)..
(4)
We then apply gradient ascent with (-) on existing designs to generate improved designs as per Eq B Although the mean ensemble generally results in superior designs compared to single proxy
3

---

Algorithm 'riimeentrrig for Offline Model-based Optimization
Input: The static dataset D. the number of iterations T, the optimizer OPT(-).
Output: The high-scoring design ii.
Initialize æ as the design with the highest score in D.
Train proxies (/()) (.)5. 10 (.) and fo (.) on DD with different initializations.
for t 0 to T do D. Voting-based pairwise supervision.
4
Sample K neighborhood points æt as S(æz))
Compute pairwise comparison labels ÿ ÿ and d for three proxies on S(x1).
Derive consensus labels: "" majority majorityy "" )²).
& Adaptive soft-labeling.
7
for proxy in If (.). f((())) ()) (o)( do
Initialize soft-labels consensus labels: n g².
Inner level: fine-tune the proxy with ().
10
Outer level: learn more accurate soft-labels ÿ" with Eq.
11
Mentor proxy using the optimized soft-labels y with Eq.
Gradient ascent with mean ensemble.
12
Form a more robust ensemble as fo(x) ( 11 + 1 (2))
13
Gradient ascent: E++1 It + nOPT (V.)o(
14 Return 2 BT
due to the ensemble robustness (valuable, albeit potentially unreliable) ranking supervision signals within every proxy. Emphasizing the relative order of designs rather than their absolute scores, these signals are more resilient to noise and inaccuracies. These ranking signals are commonly used in evolutionary algorithms [[2]. reinforcement learning [33, and generative modeling [14] to select designs and could further improve the ensemble robustness. We extract the ranking supervision signals from individual proxies in the form of pairwise comparison labels, and then combine these labels via majority voting to generate consensus labels mentor proxies. We provide detailed explanation of this module below, with its implementation shown in Algorithm [] from Line 4 to Line 6.
g 99. this approach does not fully exploit the potential of weak
Pairwise comparison label. We adopt a pairwise approach to represent the ranking supervision signals for every proxy, focusing on relative order align with the ranking information used in design selection. We sample K points at the neighborhood of the optimization point
S(x:)
=xx #x Nax,, 82) where Max, 52) represents a Gaussian distribution centered at æ, with variance 82. For each sample pair xx)) and a proxy (e.g.. fe(()) (-)), we define the pairwise comparison label y (1,66()) > fo (x))), where 1 is the indicator function. The labels ÿ from all sample pairs serves as the ranking supervision signals for the proxy fa((). We repeat this process for all proxies, generating signals j and ÿ for proxies fB (.) and fo respectively.
Majority voting. Given these pairwise comparison labels ÿ ÿd and y we derive the pairwise consensus labels ý" via an dlement-wise majority voting:
0// majority_ vootiiyoootngggggee ( 0. (
where i and j are the indexes of the neighborhood samples. As consensus labels are generally more reliable, they can be employed for mentoring the proxies to promote the exchange of ranking supervision signals. Specifically, we can fine-tune the proxy fåt (.) using the binary cross-entropy loss, where a also used in the ChatGPT reward model training [8. The loss function be computed as:
fél (x))) represents the predicted probability that fe (xx") > fg (z7).. ). as
L (A)
Σ
S loglo(dd
t6 (m"")) + 01,) (16 fo (mx))) (6)
CK
15K3S5K
where C+ = K(K-1) denotes the number of the sample pairs. This procedure is also applied to proxies (.) and (-). While our approach encourages alignment with the consensus, it does not aim to make proxies identical. Each proxy maintains its unique learning trajectory, thereby preserving the diversity among the proxies.
2

---

3.2 Adaptive Soft-labeling
The consensus labels ý' may contain noise since the majority voting consensus can be incorrect. To mitigate this issue, we propose an adaptive seft-labeling module. This module initializes soft-labels as consensus labels, and employs a bi-level optimization framework for adaptive mentoring, where the inner level fine-tunes the proxies and the outer level refines the soft-labels. We provide a detailed description of this module below; its implementation is outlined in Algorithm [
Line 7- Line 11.
Fine-tuning, We initialize the soft-labels the consensus labels: のe g², serving as an effective starting point. Utilizing the soft-labels ÿ², we can perform fine-tuning on the proxy fā against the binary cross-entropy loss following (6.
C 10, (0.0")
Σ ISiSIEK
S (5 fa (2,))) + (1 -og]) fe (2.)). (7)
C
In contrast to (q.(6). we express the loss (0, 9 as function of both the proxy parameters and the soft-labels y the accurate soft-labels ÿs have yet be determined. One-step gradient descent enables fine-tuning, resulting in the following relationship:
( - 0 OC (0.00.
(8)
00
where 7 denotes the fine-tuning learning rate.
Soft-labeling. Assuming the soft-labels are accurate, the fine-tuned proxy (.) is expected to perform well on the static dataset. This is due to the fact that, despite having different data distributions, the soft-labels and the static dataset share underlying similarities, as they both represent the same ground-truth from pairwise and pointwise perspectives, respectively. This leads to bi-level optimization framework with the fine-tuning mentioned above as the inner level and the soft-labeling here as the outer level. In particular, we enhance the accuracy of soft-labels j by minimizing the mean squared error on the static dataset D = {1)
A 0 3 n= (f e
(z.) - y1)
0 D
N
where À represents the soft-labeling learning rate. The nested optimization problem can easily solved by higher, a library for higher-order optimization [19]. Once optimized, the labels are fed back to the first module, adaptively mentoring the proxy fo' (-) according to Eq.08). The same procedure can be employed for proxies fØ²() and fg (-). ultimately resulting in a more robust ensemble. We further clarify the novelty of this module in Appendix A.2
4 Experiments
We conduct extensive experiments design-bench to investigate the effectiveness and robustness of the proposed method. In Section 4.4, we benchmark our approach against several well eellblsttbl baselines. In Section 4.5 we verify the effectiveness of two modules: voting-bass based pairwise supervision and adaptive soft-labeling, as well as other contributing factors. In Section 4.6 we investigate the sensitivity of our method to hyperparameter changes. While our primary focus is the tri tritmint with 3 proxies, we additionally explore other marttlel-menttring implementations utilizing 5, 7, 9, and 11 proxies in Appendix A.
4.1 Task Overview
We adopt the design- bench which comprises both continuous and discrete tasks. Below, we outline the dataset details and evaluation protocols.
Dataset. We conduct experiments on four continuous tasks: (1) Superconductor (SuperC) discover an 86-D superconductor to maximize critical temperature with 17010 designs. (2) Ant Morphology (Ant) 20 identify a 60-D ant morphology to crawl quickly with 10004 designs. (3) D' D'Kitty Morphology (D' (iiKtty) [21]: determine a 56-D D' Kitty morphology to crawl quickly with 10004 designs. (4) Hopper Controller (Hopper) M: find a neural network policy with 5126 weights to maximize return with 3200 designs. Besides, we perform experiments on four discrete tasks: (1)
TF Bind 8 (TFB8) [5) design a length DNA sequence to maximize binding activity score with 32896 designs. (2) TF Bind 10 (TFB10) 10) (5): find length 10 DNA sequence to maximize binding
5

---

activity score with 50000 designs. (3) NAS M: find 64-D NN with 5 categories per dimension to maximize the performance on CIFARIO with 1771 designs.
Evaluation. We use the oracle evaluation of design-bench to evaluate a certain design and the details of the oracle functions are reported in Design-Bench Benchmark Tasks in M.. Following 77), we select the top N = 128 candidates for each method and report the 100th percentile (maximum) normalized ground truth score. The score, denoted as yn is computed as V-ven
Bevin where y is the design score, and Ymiin and ymar are the lowest and highest scores in the complete unobserved dataset, respectively. In addition, we provide the 50th percentile (median) normalized ground truth scores used in the prior work in Appendix [A.3] We also provide the mean and median ranks of all comparison methods to better assess the overall performance.
4.2 Comparisons with Other Methods
In this paper, we benchmark our method against both gradient-based and non non con-gradient-based ap- proaches. The gradient-basee based methods include: (1) Grad: optimizes the design against the learned proxy via simple gradient ascent; (2) DE (Deep Ensemble) optimizes the design against the mean ensemble of three proxies via gradient ascent; (3) GB (Gradient Boosting) [2]] sequentially trains new proxies to obtain a robust proxy, followed by gradient ascent using the proxy; (4) COMs i lower bounds the proxy's prediction on out-of-distribtioon designs and subsequently carries out gradient ascent; (5) ROMA (23:: incorporates the smoothness prior into the proxy and optimizes the design against the proxy; (6) NEMO (24): leverages normalized maximum likelihood to constrain the distance between the proxy and the ground-truth, and acquires new designs by gradient ascent; (7) BDI [25): proposes to distill the information from the static dataset into the high-scoring design; (8) IOM [26:: enforces the invariance between the representations of the static dataset and generated designs to achieve a natural trade-off. Since our tri-mentoring adopts three proxies, methods using one proxy including COMs, ROMA and IOM are equipped with three proxies for fair comparison. We also explore combining our method with ROMA and COMs and please refer to Appendix [A.4 for an in-depth discussion and corresponding empirical results.
The non- gradient- based methods include: (1) BO-qEI 227: fits a Gaussian Process, proposes
candidate designs utilizing the -eass-Exppettdd Improvement acquisition function, and assigns labels to the candidates with the proxy: (2) CMA-ES (23] labels the sampled designs and gradually adapts the covariance matrix distribution towards the high-scoring part among the sampled designs; (3) REINFORCE 29 parameterizes design distribution and optimizes the distribution towards the optimal design by policy gradient; (4) CbAS 300: trains VAE model and progressively adapts the model to focus on the high-scoring designs; (5) CbA. [I:]: retrains the proxy adopted in CbAS
by leveraging importance sampling: (6) MIN [2]: learns an inverse map from a score to design
and then samples from the map conditioned an optimal score value.
4.3 Training Details
We follow the settings in 7 (25 if not specified. We adopt three-layer MLP network with the ReLU function as the activation. We train the MLP model on the static dataset with a x 10-3 learning rate and an Adam optimizer. The fine-tuning learning rate is set as 1 x 10-3 and the soft-labeling learning rate X is set as x 10-1 The standard deviation 8 is set as 1 x 10-1 and the number of the samples K is set as 10. All experiments robustness of our results, we perform 16 trials for each setting and report the mean and standard error. We'v ve detailed the training time and computational overhead of our approach in Appendix A.5 to provide a comprehensive view of its practicality.
performed
single V100 GPU. To ensure the
4.4 Results and Analysis
In Table I and Table we present the results of our experiments for continuous and discrete tasks, respectively. A delincating line is drawn to separate the gradient- based methods from the non- bradient-based methods. Results for non-gradient-ascem methods are taken from M. The highest score of the static dataset for each task is denoted by D(best). For each task, we highlight the algorithms that fall within one standard deviation of the highest performance by bolding their results.
Continuous tasks. (1) Table I demonstrates Iri-mentoring attains the best results across the board, highlighting its effectiveness. Its consistent gains over Grad confirm its ability to tackle the out- of-distribution issue. We further test our tri-mentoring for dusto--distribution issues as detailed in Appendix A.6 (2) Compared to DE and GB, which also use multiple proxies, tri-mentoring achieves

---

Table 1: Results (maximum normalized score) on continuous tasks.
Method
Superconductor
Ant Morphology
D D'Kitty Morphology
Hopper Controller
D(best)
0.399
0.565
0.884
1.000
BO-qEI
0.402 0.034
0.819 0.000
0.896 + 0.000
0.550 = 0.018
CMA-ES
0.465 + 0.024
1,214 + 0.732
0.724 + 0.001
0.604 0.215
REINFORCE
0.481 = 0.013
0.266 = 0.032
0.562 = 0.196
-0.020 0.067
CbAS
0.503 0.069
0.876 0.031
0.892 = 0.008
0.141 0.012
Auto.CbAS
0.421 1 0.045
0.882 1 0.045
0.906 1 0.006
0.137 1 0.005
MIN
0.499 1 0.017
0.445 1 0.080
0.892 1 0.011
0.424 1 0.166
Grad
0.495 0.011
0.934 0.011
0.944
1.797 = 0.116
DE
0.514 0.015
0.937 = 0.016
0.956 = 0.014
1.805 = 0.105
GB
0.496 = 0.012
0.926 = 0.029
0.948 = 0.012
1.793 = 0.429
COMs
0.491 0.028
0.856 = 0.040
0.938 0.015
0.642 = 0.167
ROMA
0.508 0.014
0.914 0.029
0.012
1.728 = 0.266
NEMO
0.502 0.002
0.958 1 0.011
0.951
0.481 + 0.003
IOM
0.522 0.018
0.926 + 0.030
0.943 0.012
1.015 + 0.380
BDI
0.513 0.000
0.906 + 0.000
0.919 + 0.000
1.993 + 0.000
Tri-mentoring
0,514 0.018
0.948 0.014
0.966 1 0.010
1,983 + 0.110
Table 2: Results (maximum normalized score) discrete tasks & ranking
all tasks.
Method
TF Bind 8
TF Bind 10
NAS
Rank Mean Rank Median
D(best)
0.439
0.467
0.436
BO-qEI
0.798 + 0.083
0.652 = 0.038
1.079 = 0.059
10.1/15
11/15
CMA-ES
0.953 + 0.022
0.670 = 0.023
0.985 1.985=001 0.079
6.4/15
4/15
REINFORCE
0.948 + 0.028
0.663 = 0.034
-1.895 = 0.000
11.4/15
15/15
CbAS
0.927 = 0.051
0.651 = 0.060
0.683 = 0.079
9.1/15
9/15
Auto.CbAS
0,910 + 0.044
0.630 0.045
0.506 0.074
11.6/15
11/15
MIN
0.905 + 0.052
0.616 + 0.021
0.717 + 0.046
11.0/15
12/15
Grad
0,886 + 0.035
0,647 = 0.021
0.624 0.102
7,9/15
9/15
DE
0.900 + 0.056
0.659 = 0.033
0.655 0.059
5.3/15
4/15
GB
0.922 1 0.050
0.630 + 0.041
0.716 0.088
7.6/15
6/15
COMs
0.496 + 0.065
0.622 + 0.003
0.783 : 0.029
10.0/15
11/15
ROMA
0.917 0.039
0.672 + 0.035
0.927
5.7/15
6/15
NEMO
0.943 1 0.005
0.711 0.021
0.737 0.010
5.0/15
4/15
TOM
0.861 0.079
0.647 + 0.027
0.559 + 0.081
7.9/15
7/15
BDI
0.870 + 0.000
0.605 + 0.000
0.722 + 0.000
8.1/15
9/15
Tri-mentoring
0.970 + 0.001
0.722 + 0.017
0.759+0.100
+ 0.102
2.1/15
2/15
better results in all four tasks, indicating its improved robustness by sharing ranking supervision signals among proxies. (3) DE typically outperforms simple gradient ascent due to ensemble prediction robustness, consistent with findings in []. (4) Other gradient-based methods, such as COMs, fail to achieve the performance as !ri-mentoring, further highlighting its superior effectiveness. (5) low-dimensional TF Bind 8 tasks, gradient- baaeed methods (average rank 8.8) underperform compared to non- eeeggrrrddntaass methods (average rank 6.8); however, in high- high-dimensional TF Bind 10 tasks, gradient- methods (average rank 7.7) surpass non- con-gradient--aad rank 8.2). This suggests non aon-graddent-basce methods, like REINFORCE and generative modeling, are more suited for oiw-dimensional design due to their global search ability, while gradient- based methods provide more direct guidance in dimenssiaani designs.
methods (average
Discrete tasks. (1) Table shows that tri-mentoring achieves the best results in two out of the three tasks, with marginal difference in the third. This indicates that tri-mentoring is a potent method for discrete tasks as well. (2) However, complex tasks such as NAS, where each design is represented as 64-1ength sequence of 5-category one-hot vectors, the performance of tri-mentoring is slightly compromised. This could be attributed to the encoding in bessign-bench not fully accounting for the sequential and hierarchical nature of network architectures, leading to less effective gradient updates. Our proposed method, also demonstrates its effectiveness on high-dimensional biological sequence design, achieving maximum normalized scores of 0.865 and 0.699 GFP and UTR respectively.
In summary, tri rri-mentoring achieves the highest ranking as shown Table 2 and Figure 3 and delivers the best performance in six out of the seven tasks.
7

---

Table 3: Ablation studies on tri tri-mentoring.
Task
tri-mentoring
w/o voting-based ps
w/o ada soft-labeling
w/o neighbor
post selection
SuperC
0.514 # 0.018
0.505 = 0.014
0.504 + 0.014
0,316 = 0.017
0.512 = 0.011
Ant
0.948 + 0.014
0.938 + 0.021
0.945 + 0.018
0.945 = 0.012
0.945 + 0.009
D'Kitty
0.966 + 0.010
0.956 + 0.010
0.947 + 0.008
0.958 + 0.008
0.970 + 0.013
Hopper
1.983 + 0.110
1.902 + 0. 138
1.916 + 108
1.839 + 0.112
1.901 + 0.148
TF Bind 8
0.970 + 0.001
0.971 = 0.003
0.944 = 0.026
0.950 = 0.018
0.949 = 0.006
TF Bind 10 NAS
0.722 + 0.017
0.694 = 0.030
0.710 + 0.025
0.643 = 0.009
0.635 = 0.027
0.759 1: 0.102
0.509 + 0.074
0.538 1: 0.082
0.606 = 0.089
0.519 + 0.076
4.5
Ablation Studies
In this subsection, the proposed method serves as the baseline, and we systematically remove each module including voting-based pairwise supervision and adaptive seft-labeling verify its contribution. The results are presented in Table [3] Besides the performance results here, we also provide evaluation of the accuracy of generated pairwise labels to further verify the effectiveness of the two modules in Appendix A.7
Voting-basee based pairwise supervision. Instead of using the proposed module, we compute the mean prediction of the ensemble and use this prediction to create pairwise consensus labels. We denote this as w/o voting- baaedd ps. Our results in Table show a decline in performance when adopting this alternative. A plausible explanation for this performance decline is that the alternative module fails to effectively exploit weak ranking supervision signals from individual proxies, resulting in reduced information exchange and collaboration among the proxies compared to the proposed module.
Adaptive soft-labeling. We remove this module and resort to using one-hot consensus labels. The performance across tasks generally deteriorates compared to the full tri-mentoring. A possible explanation for this is that this module ensures that fine-tuned proxies are optimized to perform well on the static dataset by introducing soft-labels, reducing the risk of overfitting to consensus labels derived from individual proxy predictions. This demonstrates the significance of the adaptive seft-labeling module in mitigating the effects label noise and enhancing the ensemble performance.
Furthermore, we assess the impact of neighborhood sampling in our method and post selection strategy related to our method, with results outlined in Table
Neighborhood sampling. Typically, we sample K neighborhood points at the optimization point 2, for pairwise labels. When neighborhood sampling is excluded (w/o neighbor), labels are directly generated near the static dataset. The performance generally deteriorates for w/o neighbor in Table 3 due to the lack of local ranking information around the optimization point.
Post selection. We investigate a variant, post selection, where the mean of two proxies is used to select the third proxy's 'ss candidates. From Table 3 we find that this variant generally yields worse results compared to the full tri-mentoring. This finding suggests that using the ranking supervision signals directly for design selection is less effective than allowing proxies exchange ranking supervision signals within the ensemble to produce a more robust ensemble.
4.6 Hyperparameter Sensitivity
In this section, we study the sensitivity of our method hyperparameters, specifically the number of neighborhood samples K and the number of optimization steps T, on two tasks, the continuous Ant and the discrete TFB8. We also discuss the standard deviation hyperparameter
Appendix [A.8]
Number of neighborhood samples (K). We evaluate the performance of our method for different values of K, i.c., the number of neighborhood samples around the optimization point. We test K values of 5. 10, 15, 20, and 25, with K = 10 being the default value used in this paper. The results are normalized by dividing them by the result obtained for K 10. As shown in Figure the performance of the tri-mentoring method is quite robust to changes in K for both tasks.
Number of optimization steps (T). We also investigate the impact of the number of optimization steps on the performance of our method. As indicated in Figure 5 the method is robust to changes in the number of optimization steps for both Ant and TFB8 tasks.
In summary, our sensitivity analysis demonstrates that the tri-mentoring method robust to variations in key hyperparameters, ensuring stable performance across a range of settings.

---

1.012
1.0
1.010
16 RERPONCE
Ao45,Co65 C365
Score rabio
0.8
Score
D2
1.000
TF88
col
10MA
0.995
0.6
300
10
20
25
100
200
300
400
Rank
Num of samples
Step
Figure 3: Whiskers signify rank min/max; medians and means are shown as vertical lines and black triangles, respectively.
Figure 4: The ratio of the per- formance of our tri-mentoring method with K to the perfor- mance with K 10.
Figure 5: The performance score as a function of the optimization step I' for both TFB8 and Ant tasks.
5 Related Work
Offline model-based optimization. Recently two broad groups of methods have emerged for offline bosell-baed optimization. One group is based on generative modeling, where methods aim to gradually adapt a generative model towards the high-scoring design [31-33] Another group is based on using gradient ascent to optimize existing designs via gradient information. These methods generally try to incorporate prior information into the proxy before using it for gradient ascent. Examples of this approach include COMs 77.. ROMA (33, NEMO 24), BDI 25 34 and IOM [26). Our proposed method called parallel-mentee ensemble of proxies and aim to incorporate weak ranking signals from any pair of proxies into the third. This symmetric learning process, which cycles through all proxies, enhances the robustness and resilience of our ensemble. Our proposed ensemble training process, with its focus parallel- mentoring, has the potential to improve the proxy and reward training [55 B3, thereby contributing advancements in biological sequence design. Notably, the contemporaneous ICT method [36) exchanges direct proxy scores, which may be less robust than our pairwise comparison approach.
(tri-mentoring) belongs to this category. We maintain
Tri-training. Our work is related to tri-training (37) which trains three classifiers and refines them using unlabeled samples. In each round, an unlabeled sample is labeled for one classifier the other two agree on the label, under specific conditions. While tri tri-mentoring is inspired by tri- tri-training. there are fundamental differences between them: (1) Tri-training aims to enhance classification tasks by mitigating label noise, while tri-mentoring focuses producing more robust regression ensemble; (2) Tri-training leverages unlabeled data, while tri-mentoring operates on samples near the current optimization point; (3) Tri-mentoring incorporates adaptive soft-labeling module to mitigate label noise, which is not present in tri-training. In addition to tri tri-training and tri rrimentoring.. there are other research works 38 39 40 involving multiple proxies cooperating to improve learning. Unlike tri-training and Iri-mentoring with three proxies, these methods focus on two proxies.
Bi-level optimization for hyperparameter optimization. Bi-level optimization has become increas- ingly popular for hyperparameter optimization (41]46) due to the hicrarchical problem structure [47). In the inner level, the relationship between model parameters and hyperparameters is established by minimizing the training loss. Meanwhile, the outer level optimizes hyperparameters through the connection built at the inner level by minimizing the validation loss. A specific category of hyperparameters is soft-label 488 [49], which is updated under the guidance of noise-free data to reduce noise. In this paper, propose adaptive soft-labeling reduce noise in the consensus labels.
Ensemble learning. Ensemble learning techniques train multiple base learners and aggregate their predictions to achieve better performance and generalization than individual base learners alone [8 9 50 10 II 51 52 These methods can be broadly classified into boosting [53] 54], bagging [55], and stacking [56]. In contrast to proposed tri-mentoring, where multiple proxies collaborate to enhance the learning process, ensemble learning mainly involves interaction during the aggregation phase, without influencing each other's training.
Pairwise learning to rank. Learning to rank 16 57 has been extensively employed by commercial
search engines for ranking search results 58 59 Unlike pointwise methods that score inputs, pair- wise methods focus on relative order, aligning more with ranking concepts
[57 Recent research (60) applies pairwise binary cross-entropy loss for training reward models in reinforcement learning, technique used in CharGPT [8]. Our work expresses proxy's 's ranking ability through pairwise
comparison labels, generating consensus labels via majority voting to enable mutual mentoring.

---

6 Conclusion
In this work, we introduce parallel-mentoring to enhance ensemble robustness against out-of- distribution issues through mutual mentoring among proxies. Focusing on a three-proxy case, we instantiate this as tri-mentoring, with two modules: voring-based pairwise supervision for gen- erating consensus labels, and adaptive soft-labeling which mitigates label noise through bi-level optimization. Experimental results validate our approach's is effectivenss. We discuss potential negative impacts Appendix A.9 and limitations in Appendix A.10
Acknowledgement
We thank CIFAR for support under the Al Chairs program. This research was empowered in part by the computational support provided by Compute Canada www.computecamduu.c.)
References
[1] Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-bench: Bench- marks for data-driven offline model-based optimization. arXiv preprint arXiv:2202.08450, 2022.
[2] Kam Hamidieh. data-driven statistical model for predicting the critical temperature of superconductor. Computational Materials Science, 2018.
[3] Karen S Sarkisyan et Local fitness landscape of the green fluorescent protein. Nature, 2016.
[4] Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. In Proc. Int. Conf. Learning Rep. (ICLR), 2019.
[5] Luis A Barrera et al. Survey of variation in human transcription factors reveals prevalent dna binding changes. Science, 2016.
[6] Paul Sample, Ban Wang, David W Reid, Vlad Presnyak, lain J McFadyen, David R Morris, and Georg Seelig. Human 5 UTR design and variant effect prediction from a massively parallel translation assay. Nature Biotechnology, 2019.
[7] Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Conservative objective models for effective offline model-based optimization. In Proc. Int. Conf. Learning Rep. (ICLR), 2021.
[8] Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions on pattern analysis and machine intelligence, 1990.
[9] Thomas G Dietterich. Ensemble methods in machine learning. In Multiple Classifier Systems: First International Workshop, 2000.
[10] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2017.
[11] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2019.
[12] Kevin K Yang, Zachary Wu, and Frances H Amold. Machine-learning settarninngggieeddd directed evolution for protein enginecring. Nature methods, 2019.
[13] Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. In Proc. Ini. Conf. Learning Rep. (ICLR), 2020.
10

---

[14] Alvin Chan, Ali Madani. Ben Krause, and Nikhil Naik. Deep extrapolation for attribute- enhanced generation. In Proc. Adv. Neur Inf. Proc. Syst (NeurIPS), 2021.
[15] Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: loss landscape perspective. arXiv preprint wrXiv:992.00200 2019.
[16] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 1952.
[17] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2017.
[18] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXi:22003.025555 2022.
[19] Edward Grefenstette, Brandon Amos, Denis Yarats, Phu Mon Huut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, and Soumith Chintala. Generalized inner loop meta- learning. arXiv preprint arXiv: 190.01791. 2019.
[20] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint
(66660//54)) 2016.
[21] Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. Robel: Robotics benchmarks for learning with low-cost robots. In Conf. on Robot Lea. (CoRl.), 2020.
[22] Jerome H Friedman. Greedy function approximation: gradient boosting machine. Anmals of statistics, 2001.
[23] Sihyun Yu, Sungsoo Ahn, Le Song, and Jinwoo Shin. Roma: Robust model adaptation for offline model-based optimization. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2021.
[24] Justin Fu and Sergey Levine. Offline model-based optimization via normalized maximum likelihood estimation. In Proc. Int. Conf. Learning Rep. (ICLR),
[25] Can Chen, Yingxue Zhang, Jie Fu, Xue Liu, and Mark Coates. Bidirectional learning for offline infinite-width model-based optimization. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2022.
[26] Han Qi, Yi Su, Aviral Kumar, and Sergey Levine. Data-driven mbsed optimization via
invariant representation learning. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2022.
[27] James T Wilson, Riccardo Moriconi, Frank Hutter, and Marc Peter Deisenroth. The reparame- terization trick for acquisition functions. arXiv preprint arXiv:1112.00424, 2017.
[28] Nikolaus Hansen. The ema evolution strategy: comparing review. Towards a new evolutionary computation: Advances in the estimation of distribution algorithms, 2006.
[29] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learning, 1992.
[30] David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. In Proc. Int. Conf. Machine Lea. (ICML), 2019.
[31] Clara Fannjiang and Jennifer Listgarten. Autofocused oracles for model-based design. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2020.
[32] Aviral Kumar and Sergey Levine. Model inversion networks for model-based optimization. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2020.
[33] Minsu Kim, Federico Berto, Sungsoo Ahn, and Jinkyoo Park. Bootstrapped training of score-conditioned generator for offline design of biological sequences. arXiv preprint arXiv:2306.03111,20000 2023. 2023.
11

---

[34] Can Chen, Yingxue Zhang. Xue Liu, and Mark Coates. Bidirectional learning for offline model-based biological sequence design. In Proc. Int. Conf. Machine Lea. (ICML), 2023.
[35] Moksh Jain, Emmanuel Bengio, Alex Hernandez-Garcia, Jarrid fector-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In Proc. Inf. Conf. Machine Lea. (ICML). PMLR, 2022.
[36] Ye Yuan, Can Chen, Zixuan Liu, Willie Neiswanger, and Xue Liu. Importance-awarc teaching for offline model-based optimization. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2023.
[37] Zhi-Hua Zhou and Ming Li. Tri-training: Exploiting unlabeled data using three classifiers. IEEE Transactions on knowledge and Data Engineering, 2005.
[38] Eran Malach and Shai Shalev-Shwartz. Decoupling "when to update" from "how to update". In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2017.
[39] Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the eleventh annual conference on Computational learning theory, 1998.
[40] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), volume 31, 2018.
[41] Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In Proc. Int. Conf. Machine Lea. (ICML), 2018
[42] Can Chen, Shuhao Zheng, Xi Chen, Erqun Dong, Xue Steve Liu, Hao Liu, and Dejing Dou. Generalized data weighting via class-level gradient manipulation. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2021.
[43] Can Chen, Xi Chen, Chen Ma, Zixuan Liu, and Xue Liu, Gradient- bi-level optimization for deep learning: A survey, arXiv preprint urXiv:2207.11719 2022.
[44] Can Chen, Chen Ma, Xi Chen, Sirui Song, Hao Liu, and Xue Liu. Unbiased implicit feedback via bi-level optimization. arXiv preprint arXiv:2206.001477 2022.
[45] Can Chen, Jingbo Zhou, Fan Wang, Xue Liu, and Dejing Dou. Structure-awar protein self- supervised learning. Bioinformatics, 2023.
[46] Zhixiang Chi, Li Gu, Huan Liu, Yang Wang, Yuanhao Yu, and Jin Tang. Metafscil: A meta- learning approach for few-shot class incremental learning. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recogn. (CVPR), 2022.
[47] Tommaso Giovannelli, Griffin Kent, and Luis Nunes Vicente. Bilevel optimization with multi-objective lower-level problem: Risk-neutral and risk-averse formulations. arXiv preprini arXiv:2302.05540, 2023.
[48] Yichen Wu, Jun Shu, Qi Xie, Qian Zhao, and Deyu Meng. Learning to purify noisy labels via meta soft label corrector. Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
[49] Görkem Algan and IIkay Ulusoy. Meta soft label generation for noisy labels. In 2020 25th International Conference on Pattern Recognition (ICPR), 2021.
[50] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 1503.0251, 2015.
[51] Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. In Proc. Int. Conf. Learning Rep. (ICLR), 2023.
12

---

[52] Olivier Laurent, Adrien Lafage, Enzo Tartaglione, Geoffrey Daniel, Jean mare Martinez, Andrei Bursuc, and Gianni Franchi. Packed ensembles for efficient uncertainty estimation. In Proc. Int. Conf. Learning Rep. (ICLR), 2023.
[53] Yoav Freund, Robert Schapire, and Naoki Abe. A short introduction to boosting. Journal- Japanese Society For Artificial Intelligence, 1999.
[54] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference knowledge discovery and data mining, 2016.
[55] Leo Breiman. Bagging predictors. Machine learning, 1996.
[56] David H Wolpert. Stacked generalization. Neural networks, 1992.
[57] Tie-Yan Liu et al. Learning to rank for information retrieval. Foundations and Trends@ in Information Retrieval, 2009.
[58] Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In Proc. Int. Conf. Learning Rep. (ICLR), 2005.
[59] Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. Bpr: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618, 2012.
[60] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss. Alec Radford, Dario Amodei, and Paul Christiano. Learning summarize with human feedback. In Proc. Ady. Neur. Inf. Proc. Syst (NeurIPS), 2020.
13

---

A Appendix
A.1 Scenarios for Parallel Mentoring with Multiple Proxies
A.1.1
Method
In the primary paper, we mainly focus scenario with three proxies. Here, we extend our method to incorporate M proxies. We revisit the two essential modules.
Voting-based pairwise supervision. We train M (M 2 3) parallel proxies, fo((). (.())) each initialized differently, on the static dataset. Their mean utilized as the final prediction:
(-), (-),
1 (o(+) M (o2()) +5(() +
1 (2)(
(10)
We generate the pairwise comparison labels g', .......................... and gM for each proxy in the same way. We extend the subsequent majority voting part and derive the pairwise consensus labels ÿ" via an Wlement-wise majority voting:
Vi majority_ voting (V.j. Di.
Dij!
(11)
Here, i and j are the indexes of the neighborhood samples.
Adaptive soft-labeling. This module remains the same it is designed for individual proxy. We carry out fine-tuning and soft-labeling via bi-level optimization to adaptively mentor the proxy.
Setting on M. In Eq. I.I M can be any positive number greater than reached with just two proxies. In this study, we consider M as an odd number to ensure decisive outcome in the voting process. Cases with an even number of proxies can be handled by adopting strategies like maintaining the original labels and skipping the fine-tuning step when the proxies are evenly split in their labels. However, we do not delve into these cases for brevity. We examine scenarios with equal to 3, 7, 9, and
decision may not be
A.1.2 Experiments
We conduct experiments the Ant task and the TFB8 task. The performance ratio comparing the performance of parallel mentoring to that of tri-mentoring, is computed as function of M (the number of proxies). The results are displayed in Figure 0
(1) Our observations indicate that as the number of proxies (M) increases, the performance ratios for both tasks initially improve, eventually reaching a plateau. This behavior suggests that an increased number proxies enhances the robustness of the ensemble due to the increased diversity. However, this impact lessens the number of proxies increases further, with the ensemble's robustness plateauing after a certain point. (2) Somewhat unexpectedly, the performance with M = shows a slight dip the Ant task. A possible explanation for this could be the dynamics of the voting system. When we have M = 3, some voting happens when two proxies agree but conflict with the third. However, when M increases to voting may oceur when four proxies align with another but dissent with the remaining three. Such a scenario can make consensus labels less reliable, potentially explaining the poor performance of the M case the Ant task. (3) Finally, it's important to note that adding more proxies also amplifies computational complexity. This increase could become restricting factor when trying scale the method to include a larger number of proxies.
A.2 Novelty of Adaptive Soft-labeling
Our adaptive soft-labeling approach, while bearing some superficial resemblance to existing bi-level optimization methods applied soft-labels, possesses unique characteristies that underscore its innovation. In this appendix, we detail these distinctive facets.
Contrary to previous works that aim address label noise in collected dataset, method handles label noise created by a unique voting mechanism along the optimization path.
Initialization:
Optimization: Previous work typically leverages the classification loss of a clean validation set to update the soft-label. However, our method proposes an innovative strategy of employing the regression loss of the static dataset for soft-label updates. This unique strategy is due to our novel
14

---

1.012
1.015
1.010
TF88
Ant
Performance ratio
Performance ratio
1.005
1.005
0.995
1.000
0.985
TFB8
Ant
0.995
0.975 0.975
9 Num of proxies M
0.05
0.1
0.15
0.2 0.25
Standard deviation 6
Figure 6: Ratio of performance with M prox- ies to performance with M 3 proxies.
Figure 7: Ratio of performance with standard deviation ó to performance with å = 0.10.
Table 4: Results (median normalized score) on continuous tasks.
Method
Superconductor
Ant Morphology
D D'Kitty Morphology
Hopper Controller
D(best)
0.399
0.565
0.884
1.000
BO-qEI
0.300 0.015
0.567 0.000
0.883 + 0.000
0.343 + 0.010
CMA-ES
0.379 = 0.003
-0.045 + 0.004
0.684 + 0.016
-0.033 = 0.005
REINFORCE
0.463 0.016
0.138 + 0.032
0.356 0.131
-0.064 0.003
CbAS
0.111 0.017
0.384 0.016
0.753 = 0..008
0.015 = 0.002
Auto.CbAS
0.131 = 0.010
0.364 0.014
1.06:=00022
0.019 = 0.008
MIN
0.336 = 0.016
0.618 + 0.040
0.887 + 0.004
0.352 = 0.058
Grad
0.339 0.0T5
+.564
0.007
0.384 + 0.004
DE
0.333 0.004
0.570 +
0.875 + 0.004
0.385 + 0.007
GB
0.373 = 0.013
0.550 1650:000 0.021
0.869 + 0.009
0.374 = 0.008
COMs
0.316 0.022
0.568 + 0.002
0,883 + 0.002
0.346 + 0.009
ROMA
0.368 0.019
0.856 + 0.008
0.388 + 0.007
NEMO
0.322 0.008
0.593 1 0.000
0.885 1 0.000
0.361 + 0.001
IOM
0.348 + 0.022
0.516 1 0.037
0.876 1 0.007
0.368 1 0.008
BDI
0.412 0.000
0.474 1 0.000
0.855 1 0.000
0.408 1 0.000
tri-mentoring
0.355 0.003
0.606 1 0.007
0.886 1 0.001
0.391 1 0.004
recognition that, despite their differing data distributions, the soft-labels and the static dataset share significant underlying similarities. They both represent the same ground-truth from pairwise and pointwise perspectives, respectively.
Objective: Whereas previous works primarily focus on mitigating label noise to enhance the performance of classification model, our work takes distinctive route by addressing label noise improve the performance of regression model for offline MBO. This adaptation to a new context further underscores the innovative aspects of our approach.
A.3 Additional Results 50th Percentile Scores
In the main paper, we presented the 100th percentile scores. Here, we offer supplementary results on the th percentile scores, which have been previously utilized in the design-bench work [1], to further validate the efficacy of tri-mentoring. Continuous task results can be found in Table while discrete task results and ranking statistics are shown in Table 5 A review of Table reveals that tri-mentoring achieves the highest ranking, demonstrating its effectiveness in this context.
A.4 Integration of COMs and ROMA with Tri-mentoring
We explore the integration of ROMA and COMs with tri-mentoring approach. The results in Toffer insights into the outcomes when combining these methods on the Ant and TFB8 tasks.
When COMs are combined with tri-mentoring, slight drop in performance is observed. This decrease might originate from COMs propensity to score lower for -aitt--dissriuution designs, potentially confficting with tri-mentoring's majority voting mechanism for such designs.
15

---

Table Results (median normalized score) on discrete tasks ranking on all tasks.
Method
TF Bind 8
TF Bind 10
NAS
Rank Mean
Rank Median
D(best)
0.449
0.467
0.436
BO-qEI
0.439 0.000
0.467 = 0.000
0.544 0.099
8.0/15
8/15
CMA-ES
0.484 = 0.014
0.391 =. 0.102
8.0/15
5/15
REINFORCE
0.402
0.475 0.008
-1.895
10.6/15
14/15
CbAS
0.463 0.007
0.292
12.7/15
12/15
Auto.CbAS
1 1419-00000
0.461 1 0.007
0.005
13.3/15
13/15
MIN
0.421 1 0.015
0.468 1 0.006
0.433 1 0.000
7.7/15
9/15
Grad
0.532
0.529 0.027
0.443
6.1/15
6/15
DE
0.581 0.034
0.534 = 0.014
0.474 0.085
5.4/15
4/15
GB
0,503 + 0.054
0.455 = 0.020
0.559
= 0.090
7.3/15
6/15
COMs
0,439 = 0.000
0.466 = 0.002
0.529 = 0.003
7.9/15
8/15
ROMA
0.017
0.516 = 0.020
0.529 0.008
5.7/15
5/15
NEMO
0.439 +: 0.018
0.456 + 0.015
0.568 1 0.021
7.0/15
8/15
IOM
0.437 + 0.010
0.475 + 0.010
0.083 + 0.012
9.0/15
7/15
BDI
0.439 + 0.000
0.176 0.000
0.517 + 0.000
6.6/15
7/15
tri-mentoring
0.609 1 0.021
0.527 0.008
0.516 1 0.028
3,4/15
2/15
Table 6: Integration results for tri-mentoring with COMs and ROMA.
Method
Ant
TFB8
COMs
0.856
0.496
Tri-mentoring
0.948
0.970
COMs + Tri-mentoring
0.941
0.960
ROMA
0.914
0.917
ROMA + Tri-mentoring
0.945
0.971
Conversely, ROMA inherently promotes smoother proxy models, a characteristic theoretically or- thogonal to our tri-mentoring method. The presented results hint that our current benchmarks might not be adequately challenging to fully exploit the combined strengths of ROMA and tri-mentoring.
A.5 Training Time Analysis
Understanding the time complexity and computational overhead method crucial for practical implementations. Hence, we provide detailed estimates for the training time of our approach.
For single proxy training on the static dataset, the training time required 101.1 for TFB8 and 33.5 for Ant. This training duration remains consistent across all methods employing a proxy. The added time overhead in tri-mentoring arises mainly from the proxy fine-tuning step. Specifically, the fine-tuning step takes 0.21 for TFB8 and 0.08 for Ant.
Given the minimal overhead of our method, especially when considering its benefits, believe it presents a feasible and efficient choice for researchers and practitioners aiming for effective results without compromising on speed.
A.6 Validation of Tri-mentoring against Out-of-distribution Issues
To substantiate our tri-mentoring approach's 's competence in addressing out-of-distribution (OOD) challenges, we have performed additional experiments TFBind8 and 10. We choose the two tasks as we could easily identify high-scoring designs and thereby create an out-of-distribution test set. From each dataset, sample 1000 high-scoring designs to form these, our tri meimeeno ensemble have been tested over 5 runs against a mean ensemble and a single proxy, and the mean squared error (MSE) is used the metric. These results in Table iillustrate that the tri-mentoring ensemble consistently outperforms the other models in handling OOD designs.
OOD test sets. For
A.7 Accuracy of Pairwise Consensus Labels
In addition to the performance results presented in the main paper, we also examine the accuracy of the optimized consensus labels gS' This analysis further substantiates the effectiveness of our
16

---

Table MSE Results for OOD Validation using TFBind8 and TFBind10 Datasets
Model
TFBind8 (MSE)
TFBind10 (MSE)
Tn-mentoring ensemble
0.0537 = 0.0008
0.2271 + 0.0075
Mean ensemble
0.0549 = 0.0006
0.2669 + 0.0084
Single proxy
0.0660 = 0.0012
0.2776 + 0.0185
method. For the D' D'Kitty and TFB8 tasks, we utilize the ground-truth function to determine the ground-truth pairwise labels. This enables us to assess the accuracy of g" For easier accuracy computation, these labels are converted into me- labels.
(I) Recall that the pairwise comparison labels of a single proxy serve as ranking supervision signals. In our analysis, we found that for single proxy, 13.45% of pairwise comparison labels for the D' Kitty task and 8.38% for the TFB8 task differ from the optimized consensus labels gs" This reveals the extent to which our method modifies the original labels. (2) Further analysis shows that, of the confficting optimized labels, 62.91% 91% are accurate for D Kitty and 63.16% are accurate for TFB8. These results reinforce the overall efficacy our method. (3) When we remove the voting voting-based based pairwise supervision module, we note a decrease in accuracy from 62.91% % to 52.21% for D D'Kitty and from 63. 16% to 55.63% for TFB8. Similarly, omitting the adaptive soft- soft-labeling module leads drop in accuracy from 62.91 % 57.16% for D' Kitty and from 63. 16.16% to 60.86% for TFB8. These experiments underscore the crucial role of both modules in preserving the label accuracy.
A.8 Additional Analysis on Sensitivity to the Standard Deviation Hyperparameter
We further delve into how the standard deviation hyperparameter 5 in neighborhood sampling, impacts the performance of our method. We experiment with á values of 0.05, 0.10, 0.15, 0.20, and 0.25, with 0.10 being the default value employed in this paper. The results normalized by dividing them by the result obtained for 5 = 0.10. As demonstrated Figure tri-mentoring exhibits remarkable robustness to variations in for both the continuous Ant and the discrete TFB8 tasks.
We further explore y and A. We conduct additional experiments over varied ranges on TFB8 and Ant, evaluating across the following values:
* 7: [2.5e 4, 5e- 4, le 3, 2e - 3, de - -3]
* A: [0.025, 0.05, 0.1, 0.2, 0.4]
These results in Table 8 confirm the method's 's resilience against hyperparameter variations on y and A, emphasizing its potential for real-world scenarios despite its perceived complexity.
Table 8: Sensitivity Analysis Results on TFB8 and Ant.
TFB8
Ant
Parameter
2.5c-4
5e-4
lc-3
2e-3
4e-3
2.5c-4
5e-4
le-3
2e-3
4e-3
0.903
0.995
1,000
0.996
1.006
0.953
1.008
1.000
1.021
1.022
0.945
0.983
1.000
1.000
0.995
1.011
0.989
1.000
0.992
0.997
Last but not least, we explore the number of the fine-tuning step. We select a single step primarily for efficiency and our comprehensive experiments indicate that increasing the number of fine-tuning steps does not significantly impact the model's 's performance, confirming the method's 's robustness in this regard. For the TFB8 dataset, the normalized results for 2, and fine-tuning steps are 1.000. 0.986, 1.006, and 0.992 respectively. Similarly, for the Ant dataset, they are 1 1.000, 0.994, 1.000, and 0.985. seen, the performance remains consistent across different numbers of steps.
A.9 Broader Impacts
Our work could potentially expedite the development of new materials, biomedical innovations, or robotics technologies, leading to significant advancements in these areas. However, as with all powerful tools, there are potential risks if misused. One potential negative impact could be the misuse of this technology in designing objects or entities for harmful purposes. For instance, in the wrong hands, the ability optimize designs could be used to create more efficient weapons or harmful
17

---

biological agents. Therefore, it is crucial to implement appropriate safeguards and regulations on the use of such technology, particularly in sensitive areas.
A.10 Limitations
Despite the promising results demonstrated by our method, its performance is largely dependent on the accuracy of the design encoding. For tasks high complexity, such as Neural Architecture Search (NAS) which represents each design as a 64-length sequence of 5-category one-hot vectors the performance tri-mentoring is somewhat limited. This shortfall could be due the default encoding technique of design-benn bench [1], which may fail to adequately capture the sequential and hierarchical nature of neural architectures, leading ineffective gradient updates. This challenge suggests that, while our method provides general framework for offline model-based optimization, task-specific techniques might be necessary for effective design encoding, especially in the context of complex tasks. Potential future research could explore ways of integrating problem-specife knowledge into the design encoding process to address these complexities more effectively.
18


Reviews:
[{"review_id": "cuFsaZjDds", "reviewer": "Reviewer_CizP", "summary": "The paper studies offline model-based optimization, which maximizes a black-box objective with a dataset of designs and scores. Most approaches rely on training a dataset proxy to approximate the black-box objective and perform SGD on the objective. This paper proposes using three proxies and voting-based supervision to generate labels. In addition, they use adaptive soft-labeling to make the proposed method more robust to label noises. ", "questions": "Please address the two weaknesses above.", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ A clear description of the problem background and the proposed algorithm procedure;\n+ Extensive ablation studies on various components, such as the voting based pairwise supervision;\n", "weaknesses": "- It's weak and less general that the proposed method constrains itself into the three parallel mentoring cases. I'd suggest the writing changed to a much more general form of any number of the parallel proxies; \n\n- The adaptive soft labeling seems to directly apply the meta-learning method /bi-level optimization method on the pseudo-label/soft label. The novelty seems not very justifiable. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## General Reply\n\nWe sincerely appreciate the effort and time you've invested in providing us with your insightful and constructive feedback. Your comments are invaluable to us as they offer opportunities for refining, rectifying, and enhancing the content of this paper. We are committed to meticulously revising our paper based on the suggestions you've delineated below.\n\n## Weaknesses\n\n> It's weak and less general that the proposed method constrains itself into the three parallel mentoring cases. I'd suggest the writing changed to a much more general form of any number of the parallel proxies;\n\n\nWe appreciate your feedback concerning the perceived limitation of our method to three parallel mentoring instances. However, it's important to clarify that we **have indeed considered a more general framework in our paper**, extending beyond the three-proxy scenario.\n\nAs stated in the paper's appendix under the subsection titled \"Scenarios for Parallel Mentoring with Multiple Proxies\", we have extended our method to include multiple proxies. The performance is quite robust against the number of parallel proxies.\n\nFurthermore, we **explicitly reference this aspect in three instances within the main body of our paper**:\n\n1. Lines 57-58: \"This paper primarily focuses on the three-proxy case, referred to as tri-mentoring, but we also examine the situation with more proxies in Appendix A.1.\"\n\n2. Lines 106-107: \"The method can be easily extended to incorporate more proxies, as discussed in\n107 Appendix A.1.\"\n\n3. Lines 174-175: \"While our primary focus is the tri-mentoring with 3 proxies, we additionally explore other parallel-mentoring implementations utilizing 5, 7, 9, and 11 proxies in Appendix A.1.\"\n\nTherefore, we believe that we have thoroughly addressed your concerns about the general applicability of our method, as described above. We also encourage you to refer to the global response \"Regarding Number of Parallel Proxies\".\n\n> The adaptive soft labeling seems to directly apply the meta-learning method /bi-level optimization method on the pseudo-label/soft label. The novelty seems not very justifiable.\n\nWe appreciate your thoughtful comment regarding the perceived similarity between our proposed adaptive soft labeling approach and existing bi-level optimization methods applied to soft labels[1]. However, our approach carries substantial novelty due to its fundamental differences from previous works [1] that also applied bi-level optimization to optimize soft labels. These differences lie in three key aspects: initialization, optimization, and objective.\n\n1. **Initization**: Contrary to previous works that aim to address label noise in a collected dataset, our method handles label noise created by a unique voting mechanism along the gradient optimization path. \n\n2. **Optimization**: Previous work typically leverages the classification loss of a clean validation set to update the soft label. However, our method proposes an innovative strategy of employing the regression loss of the static dataset for soft label updates. This unique strategy is due to our novel recognition that, despite their differing data distributions, the soft-labels and the static dataset share significant underlying similarities. They both represent the same ground-truth from pairwise and pointwise perspectives, respectively.\n\n3. **Objective**: Whereas previous works primarily focus on mitigating label noise to enhance the performance of a classification model, our work takes a distinctive route by addressing label noise to improve the performance of a regression model for offline MBO. This adaptation to a new context further underscores the innovative aspects of our approach.\n\nWe will incorporate these discussions into Appendix.\n\n## Overall\nDoes the provided response address your concerns? We greatly value your thorough feedback and anticipate further discussion during the rebuttal phase. Thank you.\n\n    [1] Wu Y, Shu J, Xie Q, et al. Learning to purify noisy labels via meta soft label corrector. AAAI 2021."}, {"review_id": "6nCrVtlKcT", "reviewer": "Reviewer_usfZ", "summary": "This paper introduces an innovative study that revolutionizes offline model-based optimization (offline MBO) by introducing a novel ensemble method. The proposed approach addresses a critical challenge in offline MBO, namely the handling of potentially inaccurate pseudo-labels generated by proxy models, particularly in out-of-distribution scenarios. It achieves this by harnessing the power of multiple parallel proxies, utilizing voting-based supervision, and incorporating adaptive pseudo-labeling.\n\nOne of the notable strengths of this method is its intuitive ensemble framework, which effectively mitigates the inherent inaccuracies of proxy models. Despite the technical complexity and meticulous design tailored specifically for the offline MBO task, this approach demonstrates significant performance improvements across various benchmark datasets.\n\nI recommend accepting this paper with some minor revisions.", "questions": "1. Can this method effectively solve high-dimensional biological sequence optimization problems, such as optimizing GFP and UTR sequences?\n\n2. Could you please provide an approximate estimate of the training time required for this method? This information would be valuable for future researchers interested in implementing the approach.\n\n3. Does the performance of the method depend on the number of proxies used?\n\n4. What is the rationale behind using Tri-mentoring? How would leveraging bi-mentoring between the individuals involved affect the outcomes?\n\n5. What are the expected outcomes when incorporating conservatism (e.g., COMs) or a Gaussian prior (e.g., ROMA) into the proxy model and integrating them with the tri-mentoring idea? Can these methods be orthogonal to each other?\n", "limitations": "Authors already put the limitation part in the paper which is really valuable. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "\n1. This paper presents a pioneering method that effectively mitigates the issue of proxy model inaccuracy in out-of-distribution scenarios, offering a novel solution to this longstanding challenge.\n\n2. The bi-level formulation utilized in this study demonstrates technical robustness and holds great potential for widespread application across various domains, highlighting its versatility and reliability.\n\n3. The approach of employing a three-proxy ensemble may appear simple at first glance, but its effectiveness in improving results is remarkable, showcasing the power of this streamlined yet impactful technique.\n\n4. The performance of the proposed method shows great promise, achieving impressive results not only at the 100th percentile but also at the 50th percentile, indicating consistent and reliable advancements across the entire distribution.\n\n5. The ablation study presented in this paper is exemplary in its clarity and thoroughness, providing a comprehensive analysis of the individual components and their contributions to the overall method, enhancing the scientific rigor and understanding of the approach.\n\n\n", "weaknesses": "1. The analysis lacks consideration for ensemble techniques, which could have enhanced the effectiveness of the offline MBO. Ensembles, being statistical methods, offer the opportunity for deeper analysis and intuition, especially from a statistical perspective. Incorporating such analysis and intuition into the manuscript would greatly improve its quality.\n\n2. The algorithm appears to be overly complex, introducing numerous degrees of freedom. This complexity could limit the practicality of the offline MBO approach since there is no opportunity to fine-tune hyperparameters using an oracle function. Simplifying the algorithm and reducing the number of degrees of freedom would make it more practical and applicable in real-world scenarios.\n\n3. It would be beneficial to provide a more comprehensive discussion on ensemble techniques in similar domains, such as offline biosequential design [1,2]. Exploring the applications of ensemble methods in these domains would strengthen the manuscript's relevance and provide valuable insights for researchers in related fields.\n\n[1] Kim, Minsu, et al. \"Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequences.\" arXiv preprint arXiv:2306.03111 (2023).\n\n[2] Jain, Moksh, et al. \"Biological sequence design with gflownets.\" International Conference on Machine Learning. PMLR, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## General Reply\nWe value your constructive feedback and will incorporate it diligently in our revisions. Thank you.\n## Weakness\n> The analysis lacks consideration for ensemble techniques\n\nWe thank you for highlighting the importance of ensemble. In our manuscript, we've employed and discussed two ensemble methods: Deep Ensemble (DE, mean ensemble) and Gradient Boosting (GB, sequentially trains proxies). As noted in Line 236-238, tri-mentoring outperforms DE and GB in all four tasks, demonstrating its robustness by utilizing shared ranking supervision signals.\n\nFurther, Appendix A.3 focuses on the accuracy of consensus labels generated by the three-proxy ensemble within our tri-mentoring strategy.  A majority of these labels are accurate, offering clear intuition behind the ensemble's efficacy.\n\n> The algorithm appears to be overly complex.\n\nRefer to the global response \"On Additional Hyperparameters\".\n\n> It would be beneficial to provide a more comprehensive discussion on ensemble techniques in offline biosequential design [1,2]. \n\nWe value your suggestion on delving deeper into ensemble in offline biosequential design. While our experiments already include two tasks from design-bench: TFB8 and TFB10, to design DNA, we acknowledge the significance of [1,2].\n\nYour referenced works [1,2] also use a trained proxy, which we believe can benefit from our parallel-mentoring. To underscore this, we will enhance our discussion by adding in Line 308: \"Our proposed ensemble training process, with its focus on parallel-mentoring, has the potential to improve the proxy/reward training in [1, 2], thereby contributing to advancements in biological sequence design.\" \n\n## Questions\n\n> Can this method effectively solve high-dimensional biological sequence optimization\n\nIndeed, our approach has been tested and proven effective on these tasks, achieving normalized scores of 0.865 (GFP) and 0.699 (UTR), which are quite competitive compared to performances reported in [3,4].\n\nIn the original manuscript, we choose not to include these results as suggested by [5] which states that many methods demonstrate indistinguishable performances on GFP and UTR. However, to highlight our method's versatility, we will add in Line 252: \"Our proposed method, also demonstrates its effectiveness on high-dimensional biological sequence design, achieving maximum normalized scores of 0.865 and 0.699 on GFP and UTR respectively.\"\n\n> Could you please provide an approximate estimate of the training time \n\nThank you for emphasizing the significance of training time. \n\nTo provide an approximate estimate:\n\n- Single proxy training on static dataset, the time (seconds)  is:\n  - TFB8: 101.1 s\n  - Ant: 33.5 s\n\nThis training time is consistent across all methods that utilize a proxy.\n\nAs for tri-mentoring, the added overhead primarily stems from proxy fine-tuning. Here's the breakdown:\n  - TFB8: fine-tuning step is 0.21 s.\n  - Ant: fine-tuning step is 0.08 s.\n\nThese efficient timings, coupled with our method's benefits, make our approach a practical choice for future researchers. We'll incorporate these analyses into Appendix.\n\n> Does the performance of the method depend on the number of proxies used?\n\nRefer to the global response \"Regarding Number of Parallel Proxies\".\n\n> What is the rationale behind using Tri-mentoring? \n\nThe underpinning idea behind tri-mentoring is to harness the voted ranking signal to fine-tune the proxy. This is depicted in Figure 1 of our paper. Here, we maintain three parallel proxies, namely fA, fB, and fC. Suppose we have two designs x1 and x2 near the current point. If proxies fA and fB concur that the score of x1 is larger than that of x2, while fC disagrees, we follow the majority voting principle. The agreement between fA and fB provides a more trustworthy ranking. Their voted ranking signal fV(x1) > fV(x2) is then used to mentor fC, thereby enhancing its performance.\n\nAs for bi-mentoring, it's not applicable since the majority voting we leverage necessitates at least three proxies to cast a vote, making it unfit for a bi-mentoring approach.\n\n> What are the expected outcomes when incorporating conservatism (e.g., COMs) or a Gaussian prior (e.g., ROMA)\n\nThank you for the insightful inquiry on integrating conservatism (COMs) and Gaussian priors (ROMA) with our tri-mentoring.\n\nIn brief, our additional experimental results on Ant and TFB8 tasks are:\n\n| Method | Ant | TFB8 | \n|--------|------|-----|\n| COMs | 0.856  | 0.496 |\n| Tri-mentoring | 0.948  | 0.970 |\n| COMs + Tri-mentoring | 0.941 | 0.960  |\n| ROMA | 0.914  | 0.917 |\n| Tri-mentoring | 0.948  | 0.970 |\n| ROMA + Tri-mentoring | 0.945  | 0.971 |\n\nWhile COMs combined with tri-mentoring leads to a slight drop in performance, the ROMA and tri-mentoring combination retains similar outcomes. The dip with COMs might arise from its lower scoring for out-of-distribution designs, potentially conflicting with the tri-mentoring's majority voting mechanism for ood designs.\n\nOn the other hand, ROMA, promotes smooth proxy models. Theoretically, this characteristic should be orthogonal to our tri-mentoring. The absence of improvement might suggest that our current benchmark is not sufficiently challenging to exploit the combined strengths of both ROMA and tri-mentoring. We'll ensure to incorporate these analyses into Appendix.\n\n## Overall\nDoes our response address your concerns? We look forward to continued discussion. Thank you.\n\n    [1] Kim, Minsu, et al. Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequences.\n    [2] Jain, Moksh, et al. Biological sequence design with gflownets.\n    [3] Trabucco B, Kumar A, Geng X, et al. Conservative objective models for effective offline model-based optimization.\n    [4] Chen C, Zhang Y, Fu J, et al. Bidirectional learning for offline infinite-width model-based optimization.\n    [5] Trabucco B, Geng X, Kumar A, et al. Design-bench: Benchmarks for data-driven offline model-based optimization."}, {"review_id": "gZ23iV5Nzh", "reviewer": "Reviewer_VCj9", "summary": "The aim of the paper is to tackle the out-of-distribution problem in offline model-based optimization methods. To this end, the authors drew inspiration from recent studies that proposed better ensemble proxies and weak ranking supervision signals. They proposed a new approach called parallel-mentoring, which consists of two main modules. The first module, a voting-based pairwise supervision module, generates consensus labels. The second module, an adaptive soft-labeling module, aims to learn more accurate soft labels. The authors conducted experiments on both continuous and discrete tasks, and their approach achieved the highest ranking in the results.", "questions": "1.\tHow does leveraging the knowledge of the static dataset help to learn more accurate soft-labels?", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposed a new approach, which settles the out-of-distribution problem in offline model-based optimization.\n\n2. The paper is clear and easy to understand.\n\n3. The paper conducts extensive experiments on the design-bench.\n", "weaknesses": "1.\tThe research question is poorly defined, and the statement of the importance of the research is lacking. The out-of-distribution problem is not clearly defined, and its significance is not convincingly conveyed.\n\n2.\tThere is no experimental evidence to support the assertion that the tri-mentoring approach can effectively address the out-of-distribution problem.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## General Reply\n\nWe greatly appreciate your detailed feedback and insights. We're committed to addressing each point raised thoroughly. Thank you.\n\n## Weaknesses\n\n> The research question is poorly defined, and the statement of the importance of the research is lacking. The out-of-distribution problem is not clearly defined, and its significance is not convincingly conveyed.\n\nOur sincere thanks for your insightful comments. The primary research question of our work revolves around offline model-based optimization (MBO), a topic we detail in our \"Preliminaries: Gradient Ascent on Offline Model-based Optimization\" section. It aims to maximize a black-box objective function using a static dataset of designs and scores.\n\nWe acknowledge your point regarding the necessity to highlight the importance of our research. As a remedy, we plan to emphasize this by adding the following text after the sentence \"Designing new objects or entities to optimize specific properties is a widespread challenge, encompassing various domains such as materials, robots, and DNA sequences.\" in Line 25: \"A more effective solution to this challenge holds enormous potential benefits for society, such as facilitating the proposal of new superconductors to reduce energy costs, enabling the design of faster robots for service, and advancing new drug discoveries to combat diseases.\"\n\nRegarding the out-of-distribution problem, it was our aim to illustrate its significance from Line 101 to Line 103: \"However, this method faces a challenge with the proxy being vulnerable to out-of-distribution designs. When handling designs that substantially differ from the training distribution, the proxy yields inaccurate predictions.\" We will ensure to make this point clearer in our revision to make the concept and its importance more accessible.\n\n> There is no experimental evidence to support the assertion that the tri-mentoring approach can effectively address the out-of-distribution problem.\n\nThank you for raising concerns regarding the experimental validation of our tri-mentoring approach's efficacy against out-of-distribution issues. \n\nTo directly tackle this concern, we have performed additional experiments on TFBind8 and TFBind10. We choose the two tasks as we could easily identify high-scoring designs and thereby create an out-of-distribution test set. From each dataset, we sample 1000 high-scoring designs to form our OOD test sets. For these, our tri-mentoring ensemble have been tested over 5 runs against a mean ensemble and a single proxy, and the mean squared error (MSE) is used as the metric. Here are the results:\n\n- **TFBind8**:\n  - Tri-mentoring ensemble: MSE of $0.0537 \\pm 0.0008$\n  - Mean ensemble: MSE of $0.0549 \\pm 0.0006$\n  - Single proxy: MSE of $0.0660 \\pm 0.0012$\n  \n- **TFBind10**:\n  - Tri-mentoring ensemble: MSE of $0.2271 \\pm 0.0075$\n  - Mean ensemble: MSE of $0.2669 \\pm 0.0084$\n  - Single proxy: MSE of $0.2776 \\pm 0.0185$\n\nThese results illustrate that the tri-mentoring ensemble consistently outperforms the other models in handling OOD designs. We'll incorporate these experiments and analysis into Appendix.\n\nThese findings, coupled with our main manuscript and appendices, offer a comprehensive demonstration of our method's efficacy. Specifically, in Line 234, we highlight, \"(1) Table 1 demonstrates *tri-mentoring* attains the best results across the board, highlighting its effectiveness. Its consistent gains over Grad confirm its ability to tackle the out-of-distribution issue.\" Furthermore, in Appendix A.3 \"Accuracy of Pairwise Consensus Labels,\" we illustrate that a majority of the consensus labels are accurate, underscoring our approach's competency in addressing the out-of-distribution problem.\n\nIn light of this evidence, we firmly believe in the capability of our tri-mentoring approach to effectively address the out-of-distribution problem. \n\n## Questions\n\n> How does leveraging the knowledge of the static dataset help to learn more accurate soft-labels?\n\nWe greatly appreciate your question. The primary benefit of utilizing the static dataset comes from the commonality that it shares with the soft labels. Despite having different data distributions, the soft-labels and the static dataset share underlying similarities, as they both represent the same ground-truth from pairwise and pointwise perspectives, respectively.  This inherent connection allows for a rich exchange of information between the two, significantly benefiting the learning of more accurate soft-labels.\n\nMore specifically, a proxy fine-tuned with soft-labels is expected to perform well on the static dataset due to this commonality. We thus leverage the regression loss of the static dataset to guide the optimization of soft-labels using bi-level optimization. This strategy is based on our novel recognition of the shared underlying truth between the static dataset and the soft-labels, making our method uniquely positioned to capitalize on it. The specific discussion about this methodology can be found in our manuscript from Line 158 to Line 168.\n\n## Overall\n\nHave we addressed your queries satisfactorily? Your feedback is invaluable, and we're keen on further discussions. Thank you."}, {"review_id": "9hcx84NQDP", "reviewer": "Reviewer_NEb9", "summary": "This paper introduces a novel offline model-based optimization (MBO) framework that incorporates a fine-tuning strategy for ensemble proxy models. Instead of directly using three independent proxy models, the paper suggests a \"voting\" strategy. In this scheme, based on the majority order of two (near) inputs, each model is fine-tuned using binary cross-entropy (BCE) loss before it is utilized for the optimization problem. The paper contends that this approach enables the proxy models to mentor each other, thereby leading to improved performance during the input optimization stage.", "questions": "- The paper deals with discrete tasks as well, while the optimization strategy is based on gradient ascent. How the input optimization is performed? \n- I wonder if the authors tried to incorporate the method on prior offline MBO method, e.g., COMs to pre-trained the model first and then fine-tune models via the proposed fine-tuning strategy. \n- I wonder if the model is fine-tuned sequentially or not. Specifically, once  $f_A$ is fine-tuned, then is fine-tuned $f_A$ used for fine-tuning $f_B$ or the original $f_A$ before fine-tuning used?\n- Why the fine-tuning step is set as 1? Did the authors try more steps?", "limitations": "The authors discussed the limitations and potential negative impact in their manuscript. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n- The experimental results are evaluated extensively, as well as showing quite a good performance compared with prior offline MBO methods.\n- To best of my knowledge, exploring fine-tuning strategy to improve the performance of ensemble models is relatively under-explored in offline MBO literature. In this respect, I think this paper investigates important but overlooked  problem. ", "weaknesses": "- My primary concern revolves around hyperparameter selection. It appears the method introduces a considerable number of additional hyperparameters, including, but not limited to, learning rates for fine-tuning soft-labels and proxy models, the variance $\\delta^2$ for selecting an adjacent input from the current solution, and the number of near inputs, $K$. Although the authors demonstrated the stability and effect of these hyperparameters (e.g., $K$), an analysis of some other hyperparameters, such as $\\delta^2$ or fine-tuning learning rates, is still missing. Given that the paper addresses \"offline\" MBO, the experiments should detail how these hyperparameters can be determined (e.g., $\\delta^2$ might be defined as a function of input dimension or a constant). Alternatively, the authors should demonstrate that the proposed method is robust to variations in these hyperparameters.\n- The proposed fine-tuning strategy lacks the original regression objective; the model might produce inaccurate values at training points after fine-tuning.\n- Although it's not a major weakness, any theoretical guarantee or supporting intuition would significantly enhance the manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weaknesses\n\n> My primary concern revolves around hyperparameter selection. \n\nPlease refer to the global response \"On Additional Hyperparameters\".\n\n> The proposed fine-tuning strategy lacks the original regression objective; the model might produce inaccurate values at training points after fine-tuning.\n\nThank you for pointing out the potential pitfalls of our fine-tuning strategy concerning the original regression objective. Emphasizing soft-labels can indeed shift predictions at the original training points. However, our primary goal with adaptive soft-labeling is broader: it aims to enhance the model's generalization, especially towards out-of-distribution points around the current optimization point. This might sometimes sacrifice pinpoint accuracy on training points but significantly improves the model's ranking ability and adaptability beyond the training set.\n\nTo emphasize, **our model still respects the regression objective**. The soft-labels $\\boldsymbol{\\hat{y}}^{S}$, integral to the fine-tuning, are tied to the static dataset and refined to minimize the mean squared error, as depicted in Eq.(9) of our paper:\n\n$$\\boldsymbol{\\hat{y}}^{S'} = \\boldsymbol{\\hat{y}}^{S} - \\frac{\\lambda}{N} \\frac{\\partial \\sum_{{i=1}}^{N}(f^A_{\\boldsymbol{\\theta}(\\boldsymbol{\\hat{y}})^S}(\\boldsymbol{x}_i) - y_i)^2}{\\partial (\\boldsymbol{\\hat{y}}^{S}){^\\top}}.$$\n\n\nIn short, our model finds a balance between adaptability from soft-labeling and the foundational accuracy of the original regression, resulting in a versatile and robust ensemble for real-world scenarios.\n\n> Although it's not a major weakness, any theoretical guarantee or supporting intuition would significantly enhance the manuscript.\n\nWe have illustrated the motivation of our paper in Figure 1. The underpinning idea behind tri-mentoring is to harness the voted ranking signal to fine-tune the proxy. This is clearly depicted in Figure 1 of our paper. Here, we maintain three parallel proxies, namely fA, fB, and fC. Suppose we have two designs x1 and x2 in the neighborhood of the current optimization point. If proxies fA and fB concur that the score of x1 is larger than that of x2, while fC disagrees, we follow the majority voting principle. In this scenario, the agreement between fA and fB provides a more trustworthy ranking. Their voted ranking signal fV(x1) > fV(x2) is then used to guide or 'mentor' fC, thereby enhancing its predictive performance.\n\n## Questions\n\n> The paper deals with discrete tasks as well, while the optimization strategy is based on gradient ascent. How the input optimization is performed?\n\nYou are correct to point out the discrete nature of the tasks we've addressed. For input optimization, we adopt an encoding-decoding strategy as per the design-bench [1]. This allows us to transform the discrete space into a continuous one that we can perform gradient ascent on. After obtaining optimized continuous values, they are decoded back to the original discrete space. This approach lets us effectively leverage the gradient ascent for optimization.\n\n> I wonder if the authors tried to incorporate the method on prior offline MBO method, e.g., COMs to pre-trained the model first and then fine-tune models via the proposed fine-tuning strategy.\n\nThank you for the insightful inquiry on integrating conservatism (COMs) and Gaussian priors (ROMA) with our tri-mentoring.\n\nIn brief, our additional experimental results on Ant and TFB8 tasks are:\n\n| Method | Ant | TFB8 | \n|--------|------|-----|\n| COMs | 0.856  | 0.496 |\n| Tri-mentoring | 0.948  | 0.970 |\n| COMs + Tri-mentoring | 0.941 | 0.960  |\n| ROMA | 0.914  | 0.917 |\n| Tri-mentoring | 0.948  | 0.970 |\n| ROMA + Tri-mentoring | 0.945  | 0.971 |\n\nWhile COMs combined with tri-mentoring leads to a slight drop in performance, the ROMA and tri-mentoring combination retains similar outcomes. The dip with COMs might arise from its lower scoring for out-of-distribution designs, potentially conflicting with the tri-mentoring's majority voting mechanism for odd designs.\n\nOn the other hand, ROMA, promotes smooth proxy models. Theoretically, this characteristic should be orthogonal to our tri-mentoring. The absence of improvement might suggest that our current benchmark is not sufficiently challenging to truly exploit the combined strengths of both ROMA and tri-mentoring. We'll ensure to incorporate these analyses into Appendix.\n\n> I wonder if the model is fine-tuned sequentially or not. Specifically, once fA is fine-tuned, then is fine-tuned fA used for fine-tuning fB or the original fA before fine-tuning used?\n\nAs for the fine-tuning process, it is not performed sequentially. Instead, we generate consensus labels using the three proxies, following which each proxy is independently fine-tuned using these consensus labels and the adaptive soft-labeling method. \n\n> Why the fine-tuning step is set as 1? Did the authors try more steps?\n\nThank you for raising the question regarding the number of fine-tuning steps, specifically why we have set it to 1.\n\nWe select a single step primarily for efficiency. Moreover, our comprehensive experiments indicate that increasing the number of fine-tuning steps does not significantly impact the model's performance, confirming the method's robustness in this regard. Below are our normalized results on TFB8 and Ant for 1, 2, 3, and 4 fine-tuning steps, respectively. These results are normalized by the performance with just one \n step:\n\n- TFB8: [1.000, 0.986, 1.006, 0.992]\n- Ant: [1.000, 0.994, 1.000, 0.985]\n\nAs seen, the performance remains consistent across different numbers of steps.\n\n## Overall\n\nHave we addressed your concerns adequately? Your feedback is invaluable, and we await further discussion. Thank you.\n"}]

Author rebuttal:
Dear Reviewers,

Thank you for your thorough examination of our paper and for sharing insightful feedback. We recognize your concerns and, in this rebuttal, address two main common points raised.

## On Additional Hyperparameters

> review from usfZ: The algorithm appears to be overly complex, introducing numerous degrees of freedom. This complexity could limit the practicality of the offline MBO approach since there is no opportunity to fine-tune hyperparameters using an oracle function. Simplifying the algorithm and reducing the number of degrees of freedom would make it more practical and applicable in real-world scenarios.

> review from NEb9: My primary concern revolves around hyperparameter selection. It appears the method introduces a considerable number of additional hyperparameters, including, but not limited to, learning rates for fine-tuning soft-labels and proxy models, the variance 
 for selecting an adjacent input from the current solution, and the number of near inputs,. Although the authors demonstrated the stability and effect of these hyperparameters (e.g., ), an analysis of some other hyperparameters, such as or fine-tuning learning rates, is still missing. Given that the paper addresses "offline" MBO, the experiments should detail how these hyperparameters can be determined (e.g., 
 might be defined as a function of input dimension or a constant). Alternatively, the authors should demonstrate that the proposed method is robust to variations in these hyperparameters.


We understand the concerns related to the algorithm's complexity and our focus has been on demonstrating its robustness with different hyperparameters. We have four extra introduced parameters: 1. number of samples ($K$), 2. variance $\delta$, 3. fine-tuning learning rate ($\gamma$) and 4. soft-labeling learning rate ($\lambda$). As detailed in Section 4.6 and Appendix A.4, our method remains **robust against the number of Samples ($K$) and  variance $\delta$**. As for $\gamma$ and $\lambda$, while we've set default constants (1e-3 for $\gamma$ and 1e-1 for $\lambda$), we've also conducted additional experiments over varied ranges on TFB8 and Ant:

   - Evaluated across:
     - $\gamma$: [2.5e-4, 5e-4, 1e-3, 2e-3, 4e-3]
     - $\lambda$: [0.025, 0.05, 0.1, 0.2, 0.4]

The following results are normalized by dividing them by the result obtained for the default values.

Experimental Results on TFB8
| Parameter | 2.5e-4 | 5e-4 | 1e-3 | 2e-3 | 4e-3 |
|----------|-------|------|------|------|------|
| $\gamma$ | 0.903 | 0.995 | 1.000 | 0.996 | 1.006 |
| $\lambda$ | 0.945 | 0.983 | 1.000 | 1.000 | 0.995 |

Experimental Results on Ant
| Parameter | 2.5e-4 | 5e-4 | 1e-3 | 2e-3 | 4e-3 |
|----------|-------|------|------|------|------|
| $\gamma$ | 0.953 | 1.008 | 1.000 | 1.021 | 1.022 |
| $\lambda$ | 1.011 | 0.989 | 1.000 | 0.992 | 0.997 |

These results confirm the method's **resilience against hyperparameter variations on $\gamma$ and $\lambda$**, emphasizing its potential for real-world scenarios despite its perceived complexity. We'll ensure to incorporate these analyses into Appendix A.4 for better clarity.

## Regarding Number of Parallel Proxies

> review from CizP: It's weak and less general that the proposed method constrains itself into the three parallel mentoring cases. I'd suggest the writing changed to a much more general form of any number of the parallel proxies.

> review from usfZ: Does the performance of the method depend on the number of proxies used?


Our paper indeed places an emphasis on the three-proxy configuration, commonly referred to as "tri-mentoring". However, this focus does not imply a limitation. We have actively pursued a broader framework, one that encompasses multiple proxies.

As stated in the paper's appendix under the subsection titled "Scenarios for Parallel Mentoring with Multiple Proxies", we **have extended our method to include multiple proxies**. Performance remains robust across different numbers of parallel proxies. As the number of proxies (M) increases, the performance ratios for both tasks generally improve, eventually reaching a plateau. This behavior suggests that an increased number of proxies enhances the robustness of the ensemble due to the increased diversity.

Furthermore, we **have explicitly referenced this aspect** in three instances within the main body of our paper:

1. Lines 57-58: "This paper primarily focuses on the three-proxy case, referred to as tri-mentoring, but we also examine the situation with more proxies in Appendix A.1."

2. Lines 106-107: "The method can be easily extended to incorporate more proxies, as discussed in
107 Appendix A.1."

3. Lines 174-175: "While our primary focus is the tri-mentoring with 3 proxies, we additionally explore other parallel-mentoring implementations utilizing 5, 7, 9, and 11 proxies in Appendix A.1."

Best,

Submission 51 Authors.


Meta-review: This submission introduced an interesting model-based optimization method. The novel component is coming from utilizing weak ranking supervision signals among proxies with a voting strategy. The submission suggests a novel method and also provides an extensive evaluation results. All reviewers consistently gave positive reviews with similar thoughts. We believe it is a good submission to be shared at the conference. The only major feedback is to simplify the method description to ease the unncessary complicacy.
Decision: Accept (poster)