Importance-aware Co-teaching for Offline Model-based Optimization
Authors: Ye Yuan
Can Chen
Zixuan Liu
Willie Neiswanger
Xue Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: offline model-based optimization
co-teaching
meta-learning
sample reweighting

Abstract: Offline model-based optimization aims to find a design that maximizes a property of interest using only an offline dataset, with applications in robot, protein, and molecule design, among others. A prevalent approach is gradient ascent, where a proxy model is trained on the offline dataset and then used to optimize the design. This method suffers from an out-of-distribution issue, where the proxy is not accurate for unseen designs. To mitigate this issue, we explore using a pseudo-labeler to generate valuable data for fine-tuning the proxy. 
Specifically, we propose $\textit{\textbf{I}mportance-aware \textbf{C}o-\textbf{T}eaching for Offline Model-based Optimization}~(\textbf{ICT})$. This method maintains three symmetric proxies with their mean ensemble as the final proxy, and comprises two steps. The first step is $\textit{pseudo-label-driven co-teaching}$. In this step, one proxy is iteratively selected as the pseudo-labeler for designs near the current optimization point, generating pseudo-labeled data.  Subsequently, a co-teaching process identifies small-loss samples as valuable data and exchanges them between the other two proxies for fine-tuning, promoting knowledge transfer.  This procedure is repeated three times, with a different proxy chosen as the pseudo-labeler each time, ultimately enhancing the ensemble performance.
To further improve accuracy of pseudo-labels, we perform a secondary step of $\textit{meta-learning-based sample reweighting}$,
which assigns importance weights to samples in the pseudo-labeled dataset and updates them via meta-learning. ICT achieves state-of-the-art results across multiple design-bench tasks, achieving the best mean rank $3.1$ and median rank $2$ among $15$ methods.
Our source code can be accessed here.

Importance-aware Co-teaching for Offline Model-based Optimization
Ye Yuan Can (Sam) Chen 1,2rf Zixuan Liu', Willie Neiswanger", Xue Liu'
McGill University. 2 MILA Quebec Al Institute,
University of Washington, Stanford University ye. (rtann@@mail.mmiil www.slliiuusswwwahgggtt.rrr
can. chen@mila quebec, queeec, zucksliuβcs, washingtonn edu, neiswanger@es. swwnstrd.dd..mo. edu, xueliuBcs megill.ca
Abstract
Offline model-based optimization aims to find design that maximizes property of interest using only an offline dataset, with applications in robot, protein, and molecule design, among others. A prevalent approach gradient ascent, where a proxy model is trained on the offline dataset and then used to optimize the design. This method suffers from an out-of-distribuu
issue, where the proxy not accurate for unseen designs. To mitigate this issue, we explore using pseudo- labeler to generate valuable data for fine-tuning proxy. Specifically, we propose Importance-aware Co-T Co-Teuching for Offline Model-bused Optimization (ICT). This method maintains three symmetric proxies with their mean ensemble as the final proxy, and comprises two steps. The first step is greud-label-diigg In this step, one proxy is iteratively selected as the pseudo-labeles for designs near the current optimization point, generating pseudo-labeled data. Subsequently, a co-teaching process identifies small-loss samples as valuable data and exchanges them between the other two proxies for fine-tuning, promoting knowledge transfer. This procedure is repeated three times, with a different proxy chosen as the pseudo- labeler each time, ultimately enhancing the ensemble performance. To further improve accuracy of pseudo-labels, we perform a secondary step of meta-learning- based sample reweighting, which assigns importance weights samples in the pseudo-laheled dataset and updates them via meta-learning. ICT achieves state-of- the-art results across multiple design-bench tasks, achieving the best mean rank of 3.1 and median rank of 2, among 15 methods. Our source code can be found here
co-teaching.
Introduction
primary goal in many domains to design or create new objects with desired properties 1 Examples include the design of robot morphologies (2). protein design, and molecule design 3 Numerous studies obtain new designs by iteratively querying an unknown objective function that maps design to its corresponding property score. However, in real-world scenarios, evaluating the objective function can be expensive or risky B77 As a result, is often more practical to assume access only to offline dataset of designs and their property scores. This type of problem is referred to as offline model-based optimization (MBO) [I]. The goal of MBO is to find a design that maximizes the unknown objective function using solely the offline dataset.
Equal contribution with random order. Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Gradient ascent is common approach to address the offline MBO problem. For example, as illustrated Figure 2 (a), the offline dataset may consist of three robot size and robot speed pairs P1.2.3 A simple DNN model, referred to as the vanilla proxy and represented to fit the offline dataset as an approximation to the unknown objective function. Gradient ascent subsequently applied to existing designs with respect to the vanilla proxy fo(.), aiming to generate a new design with a higher score. However, the gradient ascent method suffers from an out-of- distribution issue, where the vanilla proxy cannot accurately estimate data outside of the training distribution, leading to a significant gap between the vanilla proxy and the ground-truth function, as shown in Figure 2 (a). As consequence, the scores of new designs obtained via gradient ascent can be erroneously high [
fo(.), is trained
To mitigate the out-of-distribution issue, recent studies have suggested applying regulariza- tion techniques to either the proxy itself I+ or the design under consideration [1] These methods improve the proxy's robustness and generalization ability. However, yer un- explored approach in this domain is using a pseudo-labeler near the current point. Fine-tuning the proxy this pseudo-labeled dataset can lead to im- provement, provided that we can identify the valuable portion the pseudo-labeled dataset.
assign pseudo-labels to designs
Inspired by this, we propose Importance-awat Co-Teachingffl for Offiine Model- basee Optimiza- tion (ICT). This approach maintains three sym- metric proxies, and their mean ensemble acts as the final proxy. ICT consists of two main steps with the first step being pseudo-label-drive co-teaching as illustrated in Figure [] During this step, one proxy is iteratively selected as pseudo-labeler, followed by a co-teaching process [13] that facilitates the exchange of valuable data between the other two proxies for fine-tuning. As depicted in Figure [ there are three symmetric proxies, fo, (.), fo.(") and for (.). The entire learning cycle (the larger tri- angle) can be divided into three symmetric parts suub-rriagges), with one proxy chosen be the pseudo-labeler in turn. Taking the top triangle as an example, we select fe, as the pseudo-labeler generate pseudo labels for of points the neighborhood of the current optimization point 2t. The other two proxies, Saa{ (-) and (-), then receive the pseudo-labeled dataset. They compute the sample loss for each entry the dataset and exchange small-loss samples between them for fine-tuning. This co-teaching process encourages knowledge transfer between the two proxies, as small losses are typically indicative of valuable knowledge. The symmetric nature the three proxies allows the above process to repeat three times, with each proxy-fo. foyy-oo,C (+), fo. (-), and fo,( (ak-naging turns as the pseudo-labe labell generator. This learning cycle promotes the sharing of valuable knowl-
f
Praxy
Pseudo-labeler
Co-teauhing
Label Transfer
Learning Cyde
f2
ff
fi
fi
fg
f
<<
fz
Figure 1: label-driven co-teaching.
edge among the three symmetric proxies, allowing them to collaboratively improve the ensemble performance in handling out-of-distribun designs.
(a)
(b)
Vanilla Praxc fag(:)
Vanilla Provc fo²(.)
1.50
Paeudo-labeled Dalasel from Co teaching
We ?
Offine Dataset
Proxy foo
Fine-tuned
1:25
We No
Fine-tuned Prowy wf ICT ( (B)
Robot Speed
Rebet Speed
1.00
1.00
Pa
0.75
Pa
0.75
Pa
Truth f(-)
Truth f(-)
Pt
Meta: letzileaminng
P:
0.50
0.25
0.5
05
1.0
0.5
1.0 Robot Size
1.5
Robot Size
Figure 2: Meta-learning baasannnnng--eded sample reweighting.

---

Despite the efforts made in the first step, small-loss data may still contain inaccurate labels. During the first step, small-loss data (Pa and pt) from the pseudo-labeled dataset produced by fo. (-) are identified based on the predictions of proxy (c) and fed to proxy Soz( (-)- However, shown in Figure (a), the less accurate point ps deviates noticeably from the ground-truth, causing the fine-tuned proxy fe, (.) to diverge from the ground-truth function. To address this, we introduce the second step of ICT, metaa-eaarnngg-bate baandd sample reweighting, which aims to assign higher weights to more accurate points like Po. and lower weights to less accurate ones like pb- To accomplish this, assign an importance weight for every sample yielded by the first step (wa for pa and wa for ps) and propose meta-learning framework to update these sample weights (wa, and w;) automatically by leveraging the supervision signals from the offline dataset P1.2.3. Specifically, the proxy fine-tuned on the weighted small-loss data (Pa and pb) is expected to perform well the offline dataset, provided the weights are accurate, i.e., large Wa and small w/- We can optimize the sample weights by minimizing the loss the offline dataset Figure (b), the weight of Pa is optimized to be high, while the weight of ps is optimized to be low. Consequently, the proxy fe (-) fine-tuned on the weighted samples in Figure 2 (b) is brought closer to the ground-trun truth objective function f(-), compared to the case where the fine-tuned proxy f64) (.) is far from f()) in Figure (a). Through extensive experiments across various tasks [T) ICT proves effective at mitigating -iiitiiiiiiitnoonnnnn issues, delivering state-of- wfte-offthhe-ar results.
function of the sample weights. As illustrated in
In summary, our paper presents three main contributions:
We introduce Importance aaaararreeeeaeeeeeeeeeeeeeeeeeee aware Co- Teaching (ICT) for offline MBO. ICT consists of two steps. the greudo-label-rrvve co-teuching step, a proxy is iteratively chosen as the pseudo-labeler, initiating a co-teaching process that facilitates knowledge exchange between the other two proxies.
The second step, meta-learning-bor inaccuracies in pseudo-labels. In this step, pseudo-labeled samples are assigned importance weights, which are then optimized through meta-learning.
sumple reweighting, is introduced to alleviate potential
Extensive experiments demonstrate ICT's effectiveness in addressing yielding tfate-of-the-ee art results in multiple MBO tasks. Specifically, ICT secures the best mean rank of 3.1 and median rank of 2, among 15 methods.
issues,
2 Preliminaries
Offline model-based optimization (MBO) targets a variety of optimization problems with the goal of maximizing an unknown objective function using an offline dataset. Consider the design space X Rd, where d represents the design dimension. Formally, the offline MBO can be expressed as:
xx arg max (xx =EX'
where /()) denotes the unknown objective function, and æ E X denotes a candidate design. In this scenario, offline dataset D (( y: ].2 is available, where æ; represents specific design, such as robot size, and y, represents the corresponding score, like robot speed. In addition to robot design, similar problems also include protein and molecule design.
common strategy for tackling offline MBO involves approximating the unknown objective function f(-) using proxy function, typically represented by deep neural network (DNN) fe(+), which is trained on the offline dataset:
0" arg min
Σ (Jo(æ:) yi)²
With the trained proxy, design optimization is performed using gradient ascent steps:
xt = 2:-1 vxfo(()
for t [1,
Here, T denotes the number of steps, and signifies the learning rate. The optimal design at is acquired as Tr. This gradient ascent approach limited by an out-of- issue, as the proxy fz(x) may not accurately predict scores for unseen designs, leading to suboptimal solutions.
3

---

3 Method
In this section, we introduce Importance-awar Co-Teaching (ICT), which consists of two steps. We maintain three symmetric proxies and compute the mean ensemble as the final proxy. In See 3.1 we describe the first step, pseudo-label-driee co-teaching. This step involves iteratively selecting one proxy as the pseudo-label generator and implementing a co-teaching process to facilitate the exchange of valuable data between the remaining two proxies. Nevertheless, the samples exchanged during co- teaching might still contain inaccurate labels, which necessitates the second step meta-learning -ased sample reweighting in Sec 3.2 During this step, each sample from the previous step is assigned an importance weight and updated via meta-learning. Intuitively, the ICT process can be likened to an enhanced paper peer review procedure between three researchers preparing for submission. Each researcher, acting as an author, presents his/her paper the other two. These two serve as reviewers and co-teach each other important points to better comprehend the paper, ultimately providing their feedback to the author. A detailed depiction of the entire algorithm can be found in Algorithm] [
3.1 Pseudo-label-driven Co-teaching
Vanilla gradient ascent, expressed in Eq. 3 is prone to outtof-distributiin issues in offline model-based optimization. One potential yet unexplored solution is using a pseudo-labeler to provide pseudo-labels to designs around the optimization point. By fine-tuning the proxy using the valuable portion of the pseudo-labeled dataset, we can enhance the proxy's 's performance. To achieve this, we maintain three proxies simultaneously, computing their mean ensemble as the final proxy, and iteratively select one proxy to generate pscudo-labelle knowledge estimated have high value, by sharing small-loss data. Due to the symmetric nature of the three proxies, this process can be repeated three times for sharing valuable knowledge further.
data. The other two proxies exchange
Pseudo-label. We initially train three proxies fo, (-), fex (-), and fes (-) the whole offline dataset using Eq. (2) with different initializations, and conduct gradient ascent with their mean ensemble,
1 (f((x + fa(a.-1) + fs(a:-1)),
Ft = = 1/
where 17 is the gradient ascent learning rate. Given the current optimization point Tt, we sample Lt.M around 2, as Zt,m 2, ?e, where y is the noise coefficient and e is drawn from the standard Gaussian distribution. An alternative way is to directly sample the M points around the offline dataset, rather than the current optimization point, We detail this option in Appendix A.1 We iteratively choose one proxy, for example fe. (-). to label these points, creating pseudo- labeled dataset D1 = {(E1) fo, (x.,))) JM Lines 5 to 6 of Algorithm detail the implementation of this segment.
M points ,
...,, ,..,,...,
Co-teaching. For each sample in the pseudo-labeled dataset Dj, we compute the sample loss for fo, (.) and fo, (.). Specifically, the losses are calculated as L2, = (foz (ax,,)) - fo, (s)
and C3,5 = (Ses (x),,) - fe, (x², respectively. Small-loss samples typically contain valuable knowledge, making them ideal for enhancing proxy robustness [13]. Proxies (+) and fo, (-) then exchange the top K small-loss samples as valuable data to teach each other where K is hyperparameter. The co-teaching process enables the exchange of valuable knowledge between proxies fa, (.) and fo, (-). This part implemented as described Lines 7 to 8 of Algorithm I The symmetric design of the three proxies, fo, (+), fo, (-), and fe, (+), enables the entire process to be iterated three times with one proxy chosen as the pseudo-labeler every time.
3.2 Meta-learning-based Sample Reweighting
While the previous step effectively selects samples for fine-tuning, these samples may still contain inaccuracies. To mitigate this, we introduce a weeta-learning-basel sample reweighting step. In this step, each sample obtained from the prior step is assigned an importance weight, which is then updated using a meta-learning framework. Without loss of generality, we use fe(-) to represent any of fo. (-), fer( (-) and fe, (-) as this step applies identically all three proxies. The top K small-loss samples selected from the previous step for fine-tuning f-(-) are denoted as D. { y }
Sample Reweighting. We assign an importance weight w; to the ith selected sample and initialize these importance weights to ones. We expect smaller importance weights for less accurate samples

---

and larger importance weights for more accurate samples to improve proxy fine-tuning, With these weights, we can optimize the proxy parameters as follows:
K
0 (w) = arg min
1 0 K
Σ
( -
(5)
Since we only want to perform fine-tuning based on D., we can adopt one step of gradient descent:
K w1
{(o(2;)) of)
0 (w)
(6)
00
where o is the learning rate for fine-tuning, This part presented in Line 10 in Algorithm !
Meta-learning. The challenge now is finding a group of proper weights w. We achieve this by leveraging the supervision signals from the offline dataset, which generally accurate. If the sample weights are accurate, the proxy fine-tuned on the weighted samples is expected to perform well on the offline dataset. This is because the weighted samples aim to reflect the underlying ground-truth function that the offline dataset already captures, and both sets of data share common patterns. We optimize the sample weights by minimizing the loss of the offline dataset in meta-learning framework. The loss on the offline dataset can be written as:
1 (or -vi) N
L(O (w)) arg min
The sample weight w, for the ith sample can be updated by gradient descent:
w = wi aC(0 (w)) 00 (w)
00
ow,
(8)
a3 a (w)) (()()))) - 0,)²
K
00
00
where / is the learning rate for the meta- learning framework. From Eq. (8). it is worth mentioning that (w]) ((f represents the similarity between the gradient of the offline dataset and 00
80 the gradient of the 1th sample. This implies that a sample with gradient similar to the offline dataset will receive a higher weight and vice versa, revealing the inner mechanism of this framework. By applying the updated sample weights to Eq. (6) for fine-tuning, we improve the proxy's ' performance. This process is iteratively applied to each proxy, yielding stronger ensemble. Lines 11 13 of Algorithm showcase the execution of this part.
4 Experimental Results
4.1 Dataset and Evaluation
Dataset and Tasks. In this study, we conduct experiments on four continuous tasks and three discrete tasks. The continuous tasks include: (a) Superconductor (SuperC)(S), where the objective to develop superconductor with 86 continuous components to maximize critical temperature, using 17, 010 designs; (b) Ant Morphology (Ant) 14), where the aim to design quadrupedal ant with 60 continuous components to improve crawling speed, based on 10, 004 designs; (c) D' Kitty Morphology (D' components enhance crawling speed, using 10, 004 designs; (d) Hopper Controller (Hopper) where the aim is to identify a neural network policy with 5, 126 weights to optimize return, using 3, 200 designs. Additionally, our discrete tasks include: (e) TF Bind 8 (TF8) where the goal is to discover an 8-unit DNA sequence that maximizes binding activity score, utilizing 32, 898 designs; (f) TF Bind 10 (TF10) where the aim is to find a 10-unit DNA sequence that optimizes binding activity score, using 50, 000 designs; (g) NAS [16), where the objective is to find the optimal neural network architecture to enhance test accuracy the CIFAR-10 [17] dataset, using 1, 771 designs.
where the focus is on shaping quadrupedal D' D'Kitty with 56 continuous
Evaluation and Metrics. accordance with the evaluation protocol used in 10 we identify the top 128 designs from the offline dataset for each approach and report the th percentile normalized
5

---

Algorithm 1 Importance-aware Co-teaching
Input: Proxies Jo., fo, with parameters ,1, 82, 0g; Offline dataset D = {(x,, yi) yi)
Output: High-scoring design x"
Start the main steps of ICT */
2 2o < the design with highest score in D.
for t= .... do
4 /* Eseudo-Label- driven Co- to-teaching */
Sample M points 11,1, 21,,, ..... Ft.M around Zt.
Choose a proxy, such as fe, (.), to pseudo-label: D1 - { fo, (xc.,)) }/
Identify the top K small-loss samples in D2 using fo,() and feed them to foz (.).
8
Identify the top K small-loss samples in D1 using fe,() and feed them to fo. (-).
Meta- tearning- based Sample Le Reweight ing */
10
Initialize the sample weights w(2), w() as ones and perform fine-tuning with Eq. 6p.
11
Update (2) for fo,(.) and w(3) for fo, (-) with Eq.
12
Update 02 using updated weights (?) and Og for w with Eq. (
13
Repeat Line 6 12 with (+) Jo, (.) iteratively as pseudo-labeler in Line 6.
14
/* Finished ICT, start to op smize designs */
15
f= 2, .... do
16
Optimize 22 with Eq. 4 based on the mean ensemble of fine-tuned fer fee fos.
17 return x² -- TT
ground-truth score. This score is computed as y/ Vmaa - Visii
Mmmii where 3mnin and Ymaz represent
the minimum and maximum scores within the entire unobserved dataset, respectively. The 50th percentile (median) normalized ground-truth scores included in Appendix A.2 For better comparison, we report the best design in the offline dataset, denoted as D(best). We also provide mean and median rankings across all seven tasks for broad performance assessment.
4.2 Comparison Methods
We compare our approach with two categories of baselines: (1) those that use generative models for sampling purposes, and (2) those that apply gradient updates derived from existing designs. The generative model-based methods learn and sample from the distribution of high-scoring designs, including: (i) MIN [8], which maps scores to designs and searches this map for optimal designs; (ii) CbAS 19. which uses VAE model to adapt the design distribution towards high-scoring areas; (iii) Auto.CbAS 200, which employs importance sampling to retrain a regression model based on CbAS.
The second category encompasses: (i) Grad: carries out a basic gradient ascent on existing designs to generate new ones; (ii) Grad. Min: optimizes the lowest prediction from an ensemble of learned objective functions; (iii) Grad. Mean: optimizes the ensemble's ' mean prediction; (iv) ROMA [8]:
applies smoothness regularization on the DNN: (v) COMs 9:: uses regularization to assign lower scores to designs obtained through gradient ascent; (vi) NEMO [IO] constrains the gap between the proxy and the ground function via normalized maximum likelihood before performing gradient ascent; (vii) BDI [11 uses forward and backward mappings to distill knowledge from the offline dataset to the design; (vili) IOM 21: enforces representation invariance between the training dataset and the optimized designs.
We also compare with traditional methods in M: (i) CMA-ES 22]
gradually adjusts the distribution
towards the optimal design by modifying the covariance matrix. (ii) BO-qEI (23]: executes Bayesian Optimization to maximize the proxy, suggests designs through the quasi- skpeeted-Improveedd acquisition function, and labels the designs using the proxy function. (iii) REINFORCE [24] optimizes the distribution over the input space using the learned proxy.
4.3 Training Details
We adopt the training settings from 1 for all comparison methods unless otherwise specified. We use 3-layer MI.P (Multil.ag Layer Perceptron) with ReL.U activation for Layer
gradient updating methods, and set the hidden size to 2048. Additional hyperparameter details are elaborated in Appendix A.3

---

One of the top 128 designs from the offline dataset is iteratively selected as the starting point, as outlined in Line 2 of Algorithm We reference results from [ for non-gradient-ascees methods such as BO-qEI, CMA-ES, REINFORCE, CbAS, and Auto.ChAS. For gradient-based methods, we run each setting over trials and report the mean and standard error. All experiments are run on single NVIDIA GeForce RTX 3090 GPU.
4.4 Results and Analysis
T presents the results across different continuous domains. In all four continuous tasks, our ICT method achieves the top performance. Notably, it surpasses the basic gradient ascent, Grad, demonstrating its ability to mitigate the uut-of-distribution issue. The superior performance of Grad.mean over Grad can be attributed the ensemble model's robustness in making predictions [25]. Furthermore, ICT generally outperforms ensemble methods and other gradient-base techniques such as COMs and ROMA, demonstrating
Performance in Continuous Tasks. Table
effectiveness of our strategy. Generative model-base methods, such CbAS and MINs, however, struggle with the nigh-dimensional task Hopper Controller. Interestingly, ICT necessitates only three standard proxies and avoids the need for training generative model, which often be challenging task. These results indicate that ICT is a simple yet potent baseline for offline MBO.
Table 1: Experimental results on continuous tasks for comparison.
Method
Superconductor
Ant Morphology
D D'Kitty Morphology
Hopper Controller
D(best)
0.399
0.565
0.884
1.0
BO-qEI
0.402 + 0.034
0.819 0.000
0.896 0.000
0.550 + 0.018
CMA-ES
0.465 + 0.024
1,214 + 0.732
0.724 0.001
0.601 0.215
REINFORCE
0.481 + 0.013
0.266 + 0.032
0.562 0.196
-0.020 - 0.067
CbAS
0.503 + 0.069
0.876 + 0.031
0.892 0.008
0.141 0.012
Auto,CbAS
0.421 = 0.045
0.882 + 0.045
0.906 = 0.006
0.137 + 0.005
MIN
0.499 = 0.017
0.445 + 0.080
0.892 0.011
0.424 + 0.166
Grad
0.483 + 0.025
0.920 0.044
0.954 0.010
1.791 + 0.182
Mean
0.497 + 0.011
0.943 0.012
0.961 0.012
1.815 0.111
Min
0.505 + 0.017
0.910 + 0.038
0.936 0.006
0.543 + 0.010
COMs
0.472 0.024
0.828 + 0.034
0.913 0.023
0.658 + 0.217
ROMA
0.510 + 0.015
0.917 0.030
0.927 0.013
1.740 0.188
NEMO
0.502 1 0.002
0.952 1 0.002
0.950 0.001
0.483 1 0.005
BDI
0.513 1 0.000
0.906 1 0.000
0.919 0.000
1.993 1 0.000
IOM
0.520 1 0.018
0.918 1 0.031
0.945 0.012
1.176 1 0.452
ICT (Crr)))))
0,503 1 0.017
0.961 1 0.007
0.968 0.020
2.104 0.357
Table 2: Experimental results on discrete tasks, and ranking on all tasks for comparison.
Method
TF Bind
TF Bind TO
NAS
Rank Mean Rank Median
D(best)
0.439
0.467
0.436
BO-qEI
0.798 1 0.083
0.652 0.038
1.079 1 0.059
9.9/15
11/15
CMA-ES
0.953 1 0,022
0.670
0.023
0.985 1 0.079
6.1/15
3/15
REINFORCE
0.948 0.028
0.663
0.034
1.895 0.000
11.3/15
15/15
CbAS
0.927 0.051
0.651
0.060
0.683 0.079
9.1/15
9/15
Auto.CbAS
0.910 1 0.044
0.630
0.045
0.506 0.074
11.6/15
12/15
MIN
0.905 0.052
0.616
0.021
0.717 0.046
11.0/15
12/15
Grad
0.906 0.024
0.635
0.022
0.598
0.034
7.7/15
9/15
Mean
0.899 1 0.025
0.652
0.020
0.666 1 0.062
6.6/15
6/15
Min
0.939 1 0.013
0.638
0.029
0.001
7.3/15
8/15
COMs
0.452 0.040
0.621
0.008
0.810 0.029
10.3/15
12/15
ROMA
0.924 + 0.040
0.666
0.035
0.911 0.020
5.1/15
5/15
NEMO
0.941 0.000
0.705
0.000
0.734
5.0/15
4/15
BDI
0.870 0.000
0.605
0.000
0.722 0.000
7.9/15
8/15
10M
0.878 0.069
0.648
0.023
0.274 0.021
7.6/15
6/15
ICT (ours)
0.958 0.008
0.691
0.023
0.667 0.091
3.1/15
2/15
Performance Diserete Tasks. Table 2 showcases the outcomes across various discrete domains.
ICT attains top performances in two out of the three tasks, TF Bind 8 and TF Bind 10. These results suggest that ICT a powerful method in the discrete domain. However, in NAS, the performance of ICT is not strong, which can be attributed to two factors. Firstly, the neural network design in NAS,
7

---

represented by 64-length sequence of 5-categorical one-hot vectors, has a higher dimensionality than TF Bind and TF Bind 10, making the optimization process more complex. Furthermore, the simplistic encoding-decoding strategy in design-hench may not accurately capture the intricacies of the neural network's 's accuracy, which can only be determined after training on CIFAR1O.
Summary. ICT attains the highest rankings with a mean of 3.1/15 and median of 2/15 as shown in Table 2 and Figure B and also secures top performances in 6 out of the 7 tasks. We have further run a Welch's 's t-test between our method and the second-best method, obtaining p-values of 0.437 on SuperC. 0.004 on Ant, 0.009 on D' Kitty, 0.014 on Hopper, 0.000 on TF8, 0.045 on TF10, 0.490 on NAS. This demonstrates statistically significant improvement in 5 out of 7 tasks, reaffirming the effectiveness of our method.
4.5 Ablation Studies
To better understand the im- pact of pseudo-label-driven co-
Table Ablation studies on two core steps of ICT.
teaching (co-teaching) and meta-
Task
D
ICT
w/o co-techhhn
w/o reweighting
learning-based sample reweighting
TF8
8
0.958 1 0.008
0.905 1: 0.042
0.910 0.024
(reweighting) on the performance
TF10
10
0.691 + 0.023
0.653 + 0.018
0.654 : 0.023
NAS
64
0.667 + 0.091
0.779 + 0.071
0.666 1:200 0.090
of our proposed ICT method, we conduct ablation studies by remov- ing either co-tcaching or reweight-
SuperC
86
0.503 + 0.017
0.500 = 0.017
0.501 + 0.017
Ant
60
0.961 + 0.007
0.927 = 0.033
0.914 0.015
D' D'Kitly
56
0.968 1 0.020
0.962 1 0.021
0.959 1 0.013
ing from the full ICT approach. Ta- ble [leeepreeents the results. Beyond
Hopper
5126
2.104 1 0.357
1.453 1 0.734
1.509 1 0.166
just assessing these performance indicators, we also verify the accuracy of the samples chosen by co-teaching, as well as the efficacy of the sample weights we have calculated. We do this by referring to the ground truth, with further details provided in Appendix A.4 Our reweighting module is also compared with the recently proposed RGD method [26] as detailed in the Appendix A..3
For two of the discrete tasks (TF8 and TF10), the ICT method consistently exceeds the performance of both its ablated versions. This highlights the efficacy of the two steps when handling discrete tasks. Conversely, the exclusion of the co-teaching in NAS leads to an increase in performance. This could be attributed to the fact that the macoding-decoding strategy of NAS in design-bench is unable to accurately capture the inherent complexity of neural networks. As such, the co-teaching step, reliant on this strategy, might not be as effective. For the continuous tasks (SuperC, Ant, D' Kitty, and Hopper), observe that the complete ICT method consistently achieves superior performance. This underlines the effectiveness of the two steps when dealing with continuous tasks. The performance gains are particularly substantial in the Hopper task when the complete ICT method is compared with the ablated versions. illustrating the power of the two steps in managing nigh-dimensional continuous tasks. Overall, our ablation studies demonstrate that the inclusion of both co-teaching and reweighting in the ICT method generally enhances performance across diverse tasks and input dimensions, underscoring their integral role in our approach.
4.6 Hyperparameter Sensitivity
We first assess the robustness of our ICT method by varying the number of samples (K) selected during the co-teaching process on the continuous D' Kitty Morphology task. For this analysis, K is varied among K = 8. 16, 32, 64. In Figure 4 (a), we illustrate the 100th percentile normalized ground-truth score as a function of time step T. for each of these K values. The results demonstrate that the performance of ICT is resilient to variations in K, maintaining performances within certain range. Additionally, ICT is capable of generating high-scoring designs early on in the process, specifically achieving such designs around the time step = 50, and sustains this performance thereafter, demonstrating its robustness against the number of optimization steps T.
We further evaluate the robustness of our ICT method against the learning rate (3) for the meta- learning framework. As depicted in Figure (b), ICT's performance remains relatively consistent across a variety of 33 values, further demonstrating ICT's robustness with respect to the hyperparameter 99. We explore the fine-tuning learning rate a and conduct further experiments and analysis on TF Bind 8. Details can be found in Appendix A.6
8

---

8
69)
0.95
0.95
Score
0.90
Score
0.90
com:
0.85
K-32
K=044
0.85
(000)
0.80
0
50
100 Step T
150
200
0.80
50
100
150
200
Rank
Step T
Figure 3: The whiskers in the plot Figure 4: Ground-truth score of the design evolves as a function
mark the rank minima and max- of step T. under different values of (subplot (a)) and 3 (subplot ima, whereas the vertical lines (b)), in the D Kitty task. These results indicate that ICT attains and black triangles respectively optimal designs swiftly and maintains them stably, as well as represent the median and mean. exhibits robustness to various choices of K and B.
Related Works
Offline Model-based Optimization. Contemporary offline model-based optimization methods can be generally classified into two primary groups: (i) generating novel designs through generative models, and (ii) conducting gradient ascent on existing designs. The former methods learn and sample from the distribution of high-scoring designs including MIN (18), CbAS [19], Auto.CbAS [20] and BootGen 277. Recently, gradient-based methods have gained popularity due to their ability to leverage deep neural networks (DNNs) for improved design generation. These methods apply regularization techniques either the proxy itself 80-10 or the design under consideration [11 enhancing the proxy's 's robustness and generalization capabilities. An interesting subfield of offline MBO includes biological sequence design, which has potential applications such as designing drugs for treating diseases [27 (88). In particular, the work (27) also adopts a proxy as pseudo-labeler and aligns the generator with the proxy, technique that resonates with method. ICT falls under this category, but adopts a unique approach to improve proxy performance: it incorporates valuable knowledge from pseude-labeled dataset into other proxies for fine-tuning, thereby enhancing the ensemble performance. Notably, while the concurrent work of parallel mentoring [29] also employs pseudo-labeling, focuses on pairwise comparison labels, potentially sacrificing some information due to its discrete nature.
Sample Reweighting. Sample reweighting is commonly utilized address the issue of label noise 30 31. where each sample is assigned larger weight if it is more likely to accurate, suggest using meta-set to guide the learning of sample weights, which can enhance model training. Such an approach is grounded in a meta-learning framework which can be used to learn hyperparameters (3537 34 38/43). Inspired by distributionally robust optimization, recent work (26) proposes a re-weighted gradient descent algorithm that provides an efficient and effective means of reweighting. In this paper, the pseudo- labeled dataset generated by co-teaching may still contain some inaccuracies, while the offline dataset generally accurate. We propose a sample reweighting framework reduce the inaccuracies in the pseudo-labelee dataset by leveraging the supervision signals from the offline dataset.
using carefully designed function. Recent studies [32 34
Co-teaching. Co-teaching [3] is effective technique for mitigating label noise by leveraging insights from peer networks. It involves the concurrent training of two proxies where one proxy identifies small-loss samples within noisy mini-batch for fine-tuning the other. Co-teaching bears similarities to decoupling [4 and co-training [45). as they all involve the interaction between two models to enhance the training process. In this study, we adapt co-teaching to work with pseudo-labeled dataset generated by trained proxy, instead of relying on noisy original dataset. Specifically, we employ one proxy to select accurate samples from this pseudo-labeled dataset for fine-tuning the other, and vice versa.
6 Conclusion and Discussion
In this study, we introduce the ICT (Importance-aware Co-Teaching) method for mitigating the out-of-distribution issue prevalent in offline model-based optimization. ICT is a two-step approach. The first step pseudo-label-driver co-teaching, which iteratively selects proxy to generate pseudo- labeled data. Valuable data are identified by co-teaching to fine-tune other proxies. This process,

---

repeated three times with different pseudo-labelers facilitates knowledge transfer. In the second step, meta-learning-basee based sample reweighting assigns and updates importance weights to samples selected by the co-teaching process, further improving the proxy fine-tuning. Our experimental findings demonstrate the success of ICT. We discuss its limitations in Appendix A.7
Future Work. Though we initially design ICT with three proxies, the method's inherent scalability and flexibility make it applicable to scenarios involving N proxies. In such a scenario, we can iteratively select one proxy out of N as the pseudo-labeler to generate data. Then, each of the remaining N proxies could select small-loss samples from its perspective and provide these samples to the other N 2 proxies for fine-tuning. This process enhances knowledge transfer and facilitates cooperative learning among the proxies. Looking to the future, we plan to conduct further research into the dynamics of such an expanded ensemble of proxies.
Negative Impact. It is crucial to recognize that ICT's potential benefits come with possible negative consequences. Advanced optimization techniques can be applied for both constructive and destructive purposes, depending on their use. For example, while drug development and material design can have a positive impact on society, these techniques could also be misused to create harmful substances or products. As researchers, we must remain attentive and strive to ensure that our work is employed for the betterment of society while addressing any potential risks and ethical concerns.
7 Acknowledgement
This research was empowered in part by the computational support provided by Compute Canada www.computecanada..aa)
References
[1] Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-bench: Bench- marks for data-driven offline model-based optimization. arXiv preprint arXiv:2202.084500 2022.
[2] Thomas Liao, Grant Wang, Brian Yang. Rene Lee, Kristofer Pister, Sergey Levine, and Roberto Calandra. Data-efficient learning of morphology and controller for a microrobot. arXiv preprint 1rXiv:990..0/334, 2019.
[3] Karen S Sarkisyan et al. Local fitness landscape of the green fluorescent protein. Nature, 2016.
[4] Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. In Proc. Int. Conf. Learning Rep. (ICLR), 2019.
[5] Kam Hamidich. A data-driven statistical model for predicting the critical temperature of superconductor. Computational Materials Science, 2018.
[6] Luis A Barrera et al. Survey of variation in human transcription factors reveals prevalent dna binding changes. Science, 2016.
[7] Paul J Sample, Ban Wang, David W Reid, Vlad Presnyak, Iain J McFadyen, David R Morris, and Georg Seelig. Human 5 UTR design and variant effect prediction from a massively parallel translation assay. Nature Biotechnology, 2019.
[8] Sihyun Yu, Sungsoo Ahn, Le Song, and Jinwoo Shin. Roma: Robust model adaptation for offline mosel-based optimization. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2021.
[9] Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Conservative objective models for effective offline model-based optimization. In Proc. Int. Conf. Learning Rep. (ICLR), 2021.
[10] Justin Fu and Sergey Levine. Offline model-based optimization via normalized maximum likelihood estimation. Proc. Int. Conf. Learning Rep. (ICLR), 2021.
[11] Can Chen, Yingxue Zhang, Jie Fu, Xue Liu, and Mark Coates. Bidirectional learning for offline infinite-wii width model-based optimization. In Proc. Adv. Neur. Inf. Proc. Syst (NeurlPS), 2022.
10

---

[12] Can Chen, Yingxue Zhang. Xue Liu, and Mark Coates. Bidirectional learning for offline model-based biological sequence design. In Proc. Int. Conf. Machine Lea. (ICML), 2023.
[13] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2018.
[14] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint (rXXiv:60000154000 2016.
[15] Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. Robel: Robotics benchmarks for learning with low-cost robots. In Conf. on Robot Lea. (CoRL), 2020.
[16] Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning, arXiv preprint arXiv: 16/ 1.01578, 2017.
[17] Geoffrey F. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhut- dinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv: 1ew.111000000 000000050,0000 2012.
[18] Aviral Kumar and Sergey Levine. Model inversion networks for model-based optimization. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2020.
[19] David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. In Proc. Int. Conf. Machine Lea. (ICML), 2019.
[20] Clara Fannjiang and Jennifer Listgarten. Autofocused oracles for model-basee design. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2020.
[21] Han Qi, Yi Su, Aviral Kumar, and Sergey Levine. Data-driven model-based optimization via invariant representation learning. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2022.
[22] Nikolaus Hansen. The CMA evolution strategy: A comparing review. In Towards New Evolutionary Computation: Advances in the Estimation of Distribution Algorithms, 2006.
[23] James T Wilson, Riccardo Moriconi, Frank Hutter, and Marc Peter Deisenroth. The reparame- terization trick for acquisition functions. arXiv preprint arXiv:1712.00424, 2017.
[24] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learning, 1992.
[25] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sehastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2019.
[26] Ramnath Kumar, Kushal Majmundar, Dheeraj Nagaraj, and Arun Sai Suggala. Stochas- tic re-weighted gradient descent via distributionally robust optimization. arXiv preprint arXiv:2306.09222, 2023.
[27] Minsu Kim, Federico Berto, Sungsoo Ahn, and Jinkyoo Park. Bootstrapped training of score-conditioned generator for offline design of biological sequences. arXiv preprint arXiv:2306.03111, 2023.
[28] Moksh Jain, Emmanuel Bengio, Alex Hermandez- Garcia, Jarrid Rector- Brotor--rooos,, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In Proc. Int. Conf. Machine Lea. (ICML).. PMLR, 2022.
[29] Can Chen, Christopher Beckham, Zixuan Liu, Xue Liu, and Christopher Pal. Parntlel-mentoring for offline model-based optimization. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2023.
11

---

[30] Lu Jiang, Deyu Meng, Teruko Mitamura, and Alexander G Hauptmann. Easy samples first: Self self-paced reranking for zero-example multimedia search. In Proceedings of the 22nd ACM international conference on Multimedia, 2014.
[31] Yixin Wang, Alp Kucukelbir, and David M Blei. Robust probabilistic modeling with bayesian data reweighting. In Proc. Int. Conf. Machine Lea. (ICML), 2017.
[32] Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In Proc. Int. Conf. Machine Lea. (ICML), 2018.
[33] Jun Shu, Qi Xie, Lixuan Yi. Qian Zhao. Sanping Zhou, Zongben Xu, and Deyu Meng, Meta- weight-net: Learning an explicit mapping for sample weighting. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2019.
[34] Can Chen, Shuhao Zheng, Xi Chen, Erqun Dong, Xue Steve Liu, Hao Liu, and Dejing Dou. Generalized data weighting via class-level gradient manipulation. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2021.
[35] Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In Proc. Int. Conf. Machine Lea. (ICML), 2018.
[36] Can Chen, Xi Chen, Chen Ma, Zixuan Liu, and Xue Liu. Gradient-based bi-level optimization for deep learning: A survey. arXiv preprint urXiv:2207.117199 2022.
[37] Fuyuan Lyu, Xing Tang, Huifeng Guo, Ruiming Tang, Xiuqiang He, Rui Zhang, and Xue Liu. Memorize, factorize, or be naive: Learning optimal feature interaction methods for ctr prediction. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), 2022.
[38] Can Chen, Chen Ma, Xi Chen, Sirui Song. Hao Liu, and Xue Liu. Unbiased implicit feedback via bi-level optimization. arXiv preprint arXiv:2206.0011 2022.
[39] Can Chen, Jingbo Zhou, Fan Wang, Xue Liu, and Dejing Dou. Structure-awar protein self- supervised learning. Bioinformatics, 2023.
[40] Tommaso Giovannelli, Griffin Kent, and Luis Nunes Vicente. Bilevel optimization with a multi-objective lower-level problem: Risk-neutral and risk-averse formulations. arXiv preprint arXiv:2302.05540, 2023.
[41] Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama. Dataset distillation using parameter pruning. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, 2023.
[42] Zhixiang Chi, Yang Wang, Yuanhao Yu, and Jin Tang. Test-time fast adaptation for dynamic scene deblurring via mexi-auxiliary learning. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recogn. (CVPR), 2021.
[43] Zhixiang Chi, Li Gu, Huan Liu, Yang Wang, Yuanhao Yu, and Jin Tang. Metafscil: A meta- learning approach for few-shot class incremental learning. In Proc. IEEE/CVF Conf. Comput. Vis. Pallern Recogn. (CVPR), 2022.
[44] Eran Malach and Shai Shalev-Shwarte Decoupling" when update" from" how to update". Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS), 2017.
[45] Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the eleventh annual conference on Computational learning theory, 1998.
[46] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. Int. Conf. Learning Rep. (ICLR), 2015.
12

---

A
Appendix
A.1
Considerations for Sampling Around the Offiie Dataset
In this subsection, we explore an alternative sampling strategy for the pseudo process. Instead of generating new samples around the current optimization point, this strategy generates samples directly around the offline dataset D. To ascertain the effectiveness of our chosen strategy against this alternative, we perform experiments on two tasks: D' Kitty (continuous) and TF8 (discrete).
Table A showcases the results. For both tasks, our strategy consistently yields higher scores, affirming its superior performance over the alternative. The advantage of our chosen strategy can be attributed to its dynamic nature. By sampling around the current optimization point, we gather more insightful information for the local fine-tuning of the proxy. This strategy allows the co-feaching process to adapt and evolve according to the optimization trajectory, leading to improved performances.
Table Comparison of Sampling Strategies. Method Sampling along Gradient Path (Ours) Sampling from 2
TFS
0.958 0.008
0.871 + 0.067
D'Kitty
0.968 0.020
0.955 + 0.006
A.2 Comparative Performance Analysis using Median Scores
In addition to the maximum scores discussed in the main paper, we also present the median (50th percentile) scores across all seven tasks. The best design in the offline dataset, denoted D(best), along with the mean and median rankings are provided for comprehensive comparison.
Performance in Continuous Tasks. fable[Sjilussratee the performances of ICT compared with other methods in continuous tasks. noteworthy that ICT exhibits performance on par with the best- performing methods. Compared with the vanilla gradient ascent (Grad), ICT demonstrates superior performance, thus affirming its effectiveness in addressing out-of-distribution issues. Moreover, ICT is generally better than the mean ensemble (Mean), which demonstrates the effectiveness of our strategy. These results support the use of ICT as a robust baseline for offline MBO.
Table 5: Experimental results continuous tasks for comparison (median).
Method
Superconductor
Ant Morphology
D'Kitty Morphology
Hopper Controller
D(best)
0.399
0.565
0.881
1.0
BO-qEI
0.300 + 0.015
0.567 + 0.000
0.883 0.000
0.343 + 0.010
CMA-ES
0.379 + 0.003
-0.045 0.004
0.684 0.016
-0.033 0.005
REINFORCE
0.463 : 0.016
0.138 = 0.032
0.356 0.131
-0.064 = 0.003
CbAS
0.111 = 0.017
0.384 0.016
0.753 0.008
0.015 0.002
Auto.CbAS
0.131 = 0.010
0.364 0.014
0.736 0.025
0.019 0.008
MIN
0.336 1 0.016
0.618 1 0.040
0.887 0.004
0.352 1 0.058
Grad
0.321 + 0.010
0.559 + 0.032
0.856 0.009
0.354 + 0.OTO
Mean
+.04
0.569 + 0.010
0.876 0.003
0.386 0.003
Min
0.354 0.026
0.571 0.011
0.883 0.000
0.359 0.004
COMs
0.316 + 0.026
0.560 + 0.002
0.879 0.002
0.341 1 0.009
ROMA
0.372 1 0.019
0.479 1 0.041
0.853 0.007
0.389 1 0.005
NEMO
0.318 1 0.008
0.592 0.000
0.880 0.000
0.355 0.002
BDI
0.412 1 0.000
0.474 1 0.000
0.855 0.000
0.408 0.000
10M
0.352 1 0.021
0.509 1 0.033
0.876 0.006
0.370 1 0.009
ICT (ours)
0.399 1 0.012
0.592 0.025
0.874 0.005
0.362 0.004
Performance in Discrete Tasks. The median scores for discrete tasks are reported in Table ICT consistently demonstrates high performance for both TF Bind 8 and TF Bind 10. However, for the NAS task, which has a higher dimensionality than the two tasks, the optimization process becomes notably more complex. Further, the simplistic design bench may not accurately capture the intricacies of the neural network's accuracy, potentially contributing to ICT's suboptimal performance on the NAS task.
-deooiin strategy employed in the
13

---

Table 6: Experimental results on discrete tasks & ranking on all tasks for comparison (median).
Method
TF Bind 8
TF Bind 10
NAS
Rank Mean
Rank Median
D(best)
0.439
0.467
0.436
BO-qEI
0.439 = 0.000
0.467 + 0.000
15049900099
7.7/15
8/15
CMA-ES
0.537 = 0.014
0.484 + 0.014
0.591 =
8,4/15
6/15
REINFORCE
0.462 = 0.021
0.475 0.008
-1.895
10.9/15
14/15
CbAS
0.428 = 0.010
0.463 = 0.007
0.292
0.027
12.9/15
13/15
Auto.CbAS
0.419 1 0.007
0.461 0.007
0.217 0.005
13.4/15
13/15
MIN
0.421 1 0.015
0.468 1 0.006
0.433 1 0.000
7.7/15
9/15
Grad
0.528 0.021
0.519 0.017
0.438 0.110
7.7/15
8/15
Mcan
0.539 = 0.030
0.539 = 0.010
0.494 = 0.077
5.3/15
5/15
Min
0.569 = 0.050
0.485 + 0.021
0.567 = 0.006
3.7/15
4/15
COMs
0.439 0.000
0.467 0.002
0.325 = 0.003
8.4/15
8/15
ROMA
0.555 = 0.020
0.512 0.020
0.525 = 0.003
5.6/15
5/15
NEMO
0.438 = 0.001
0.454 0.001
0.564 1 0.016
7.7/15
7/15
BDI
0.439 = 0.000
0.476
0.000
0.517 + 0.000
6.7/15
8/15
TOM
0.439
0.000
0.477 + 0.010
0.050
7.9/15
7/15
ICT (ours)
0,551 0.013
0.541 0.004
0.491 0.091
1.3/15
3/15
Summary. ICT excels by achieving the best median ranking and top-two mean ranking. These rankings consolidate ICT's standing as a strong method for both continuous and discrete tasks.
A.3 Hyperparameter Setting
We report the details of hyperparameters our experiments. The number of iterations, T, set to 200 for continuous tasks and 100 for discrete tasks. For most continuous and discrete tasks, we employ the Adam optimizer [46) to fine-tune the proxies. The learning rates are set at 1e 3 and le - for continuous tasks and discrete tasks, respectively. In the case of the Hopper Controller task, the input dimension is significantly larger, at 5126, and we adopt a smaller learning rate le - for fine-tuning to ensure stability of the optimization process. Regarding the learning rate for the meta-learning framework, we use the Adam optimizer [46] with a learning rate 2e for continuous tasks and 3e 1 for discrete tasks, respectively.
A.4 Analysis of Co-teaching and Sample Reweighting Efficacy
In our analysis, we focus on two key steps of our method: (1) pseudo-label-drrven co-teaching and (2) meta-learning-basse sample reweighting. We evaluate the efficacy these steps by comparing generated samples with their corresponding ground truth. It's important note that during the training phase, ground-trum -rutt scores are inaccessible to all algorithms and are used strictly for evaluation. Our method incorporates three proxies fe. (-), fe. (-), and fe, (-). We employ fe, (-) for pseudo-labeling and fo. (.), fos (-) for co-teaching. We run ICT over 50 time steps for both Kitty (continuous) and TF8 (discrete) tasks.
co-teaching. The step involves selecting 64 samples with smaller losses for fine-tuning the proxies while ignoring the remaining 64 samples. To assess the effectiveness of this strategy, we calculate S the mean squared error (MSE) between the pseudo-labeled and ground truth scores of the selected 64 samples, and L the MSE for the ignored samples. These calculations are averaged over 50 steps. We find that for D' Ditttty, SSe is 0.124 lower than L'gn and for TFS, it's 0.006 less than Llsn, These results validate the efficacy of this step, the selected samples more closely align with the ground truth.
Pseudo-label-drivee
Meta-learning-bawee based sample reweighting, In this step, we aim to assign larger weights to cleaner samples and smaller weights to noisier ones among the total of 64 samples. We measure the efficacy of this step by calculating Laarge the MSE between the pseudo-labeled and ground-trutt scores of 32 samples with larger weights, and (Small, the MSE for the 32 samples with smaller weights. These calculations are averaged over 50 steps. We observe that for D Kitty, Laarge is 0.010 lower than Ll For TF8, LLorge 0.005 less than [Smell These findings indicate that the samples with larger weights indeed closer to the ground truth, substantiating the effectiveness of this step.
14

---

A.5 Comparison with RGD
We conduct a deeper analysis comparing our sample reweighting method with the RGD method, as proposed in [26]. The RGD method, drawing inspiration from distributionally robust optimization, aims to provide an efficient and theoretically sound way to reweight samples during training.
To ensure fair comparison, perform additional experiments. keeping the dataset and experimental conditions consistent for both methods. Two benchmarks, TFB8 and D Kitty, are evaluated.
Table 7: Comparison between Sample Reweighting (Ours) and RGD-EXP.
Method
TFB8
D'Kitty
Sample Reweighting (Ours)
0.958 + 0.008
0.968 + 0.020
RGD-EXP
0.906 + 0.058
0.955 = 0.014
Our results in Table indicate that while RGD efficient and provides a theoretically effective solution, our approach delivers slightly superior performance. This performance edge might stem from our method's capability to utilize information from the static dataset.
A.6
Examining Hyperparameter Sensitivity Further
(a)
(b)
0.90
0.90
0.80
0.80
K=64
A=4e-1
8=2e-1
A=8n-2
0=5e-2
K=32
A-10-1
0.60
0.60
0.50
0.50
0
25
50
75
100
100
Step T
Step T
(c)
(d)
0.95
0.80
Score
a=de-3
Score
dm44-3
0=8e-1
0.70
2=2e:3
5e-4
0.90
d=2e-3
a=5e-4
d-10-3
0-1e-3
0.60
0.85
0.50
0
25
50
75
100
0.80
0
50
100
150
200
Step T
Step
Figure 5: Extended Analysis on Hyperparameter Sensitivity.
we delve deeper into hyperparameter sensitivity, focusing on the TF8 task. Specifically, we investigate the influence of the number of selected samples (K) in the first step, and the learning rate (5) in the second step.
Building on the analysis from Sec 4.6
Figure 5 (a) displays the 100th percentile normalized ground ground-truth truth score function of the time step T for different K values (8. 16. 32, 64). ICT demonstrates stability over specific range for varying K values, showcasing its robustness. Notably, ICT reaches optimal designs around = 20 and maintains this level, further validating its resilience against different optimization steps T.
Figure 5 (b) plots the 100th percentile normalized ground-truth score as function of the learning rate (3) in TF8. ICT maintains a consistent performance across diverse 3 values, corroborating its robustness concerning the hyperparameter 3 in TF8.
Furthermore, we evaluate the effect of the fine-tuning learning rate in both TF8 and D Kitttty tasks. Figures (c) (d) reveal a consistent performance across varied values for both tasks, highlighting ICT's robustness towards the fine-tuning learning rate.
15

---

A.7 Limitation
We validate the effectiveness of ICT across broad spectrum tasks, Nevertheless, certain evaluation methodologies do not completely represent authentic situations. For instance, in the superconductor task [5], we adhere the established convention of utilizing random forest regression model as the oracle, in line with previous studies []]. Regrettably, this model may not perfectly mirror the complexities of real-world cases, resulting discrepancies between our oracle and the ground-truth. Future collaborations with domain experts can potentially refine these evaluation methods. Overall, given the straightforward formulation of ICT, combined with empirical proof of its robustness and effectiveness across diverse tasks in the design-bench [[], we maintain confidence in its capability to effectively generalize to other scenarios.
16


Reviews:
[{"review_id": "QeQl401Cc6", "reviewer": "Reviewer_j29o", "summary": "The work proposes an Importance-aware Co-Teaching for Offline Model-based Optimization (ICT) as a solution for Offline model-based optimization problems. The ICT algorithm is comprised of 2 steps: (i) pseudo-label-driven co-teaching and (ii) per-sample reweighting.\n\nOverall the approach aims to learn an ensemble of methods where the targets predicted by one is treated as the pseudo-label for the other models. Furthermore, the more confident samples with low losses are used to give feedback to the proxy model.", "questions": "I am interested in understand two design choices taken in this work:\n\n* In co-teaching, the two models other than proxy share knowledge about top-K points which both have learnt very well. If the K points are in-distribution and all models have learnt them perfectly, I see little to gain from this step. I was thinking using the worst-K points might be more interesting, as I am not sure how the training dynamics would play out there.\n* I per-sample weighting, i am curious how the current proposed regime fares compared to other solutions such as RGD [2] which have been shown to achieve improved generalization inspired by DRO literature. RGD would also remove the need to learn another meta-network  However, this again focusses on giving more importance to samples the model is weak at, and roughly connects back to the first point.\n\n[2] Kumar, Ramnath, et al. \"Stochastic Re-weighted Gradient Descent via Distributionally Robust Optimization.\" arXiv preprint arXiv:2306.09222 (2023).", "limitations": "The approach is in the domain of optimization, and does not directly have any negative impact by itself. The authors have provided a nice paragraph covering this aspect in their paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper proposes their intuition backed by two clearly laid out steps to walk the readers through the working of the model.\n* The extensive results on multiple datasets of both continuous and discrete domains help better bolster their approach.\n* The ablation of each step is useful to better weigh each design choice of ICT", "weaknesses": "* Some of the results in Table 1, Table 2 are within standard deviation. I would advice the authors to also perform a p-test to better strengthen their claims.\n* It would be nice to expand more on the per-sample weighting part. My knowledge on this subject often uses a validation data to as a proxy for minimizing loss, similar to Meta-weight-Net [1]. In the current training regime, it is not clear to me why the model cannot converge to the trivial solution of a zero vector as per-sample weights.\n\n[1] Shu, Jun, et al. \"Meta-weight-net: Learning an explicit mapping for sample weighting.\" Advances in neural information processing systems 32 (2019).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## General Reply\n\nThank you for your insightful and constructive feedback! We commit to attentively revising the manuscript based on your remarks, as we describe in detail below.\n\n## Weakness\n\n>  I would advice the authors to also perform a p-test to better strengthen their claims.\n \nWe appreciate your advice to perform a p-test. We have run a statistical significance test (Welch's t-test) between our method and the second-best method. The p-values are 0.437 on SuperC, 0.004 on Ant, 0.009 on D'Kitty, 0.014 on Hopper, 0.000 on TF8, 0.045 on TF10, 0.490 on NAS. This demonstrates our method has statistically significant improvement over other methods in a majority of tasks (5/7), further corroborating the assertions made in our manuscript. We will add the following sentence in Line 257: \" We have further run a Welch's t-test between our method and the second-best method, obtaining p-values of 0.437 on SuperC, 0.004 on Ant, 0.009 on D'Kitty, 0.014 on Hopper, 0.000 on TF8, 0.045 on TF10, 0.490 on NAS. This demonstrates statistically significant improvement in 5 out of 7 tasks, reaffirming the effectiveness of our method.\"\n \n> It would be nice to expand more on the per-sample weighting part. In the current training regime, it is not clear to me why the model cannot converge to the trivial solution of a zero vector as per-sample weights.\n\nThank you for your feedback. As you noted, methods such as Meta-weight-Net [1] leverage validation data to guide the optimization of sample weights within the training set. Similarly, our approach employs the offline dataset to optimize the sample weights of the pseudo-labeled data. Here the offline dataset plays a similar role as the validation data.\n\nTo address your concern about the potential for convergence to a trivial solution of zero for the per-sample weights, we would like to clarify that the weights aren't being updated through the direct optimization of the weighted sample loss in Eq. (5):\n$$\n\\boldsymbol{\\theta}^{*}(\\boldsymbol{\\omega}) = \\arg\\min_{\\boldsymbol{\\theta}}\\frac{1}{K} \\sum_{i=1}^{K} \\boldsymbol{\\omega_i} (f_{\\boldsymbol{\\theta}}(\\boldsymbol{x}^s_i) - \\bar{y}^s_i)^2.\n$$\nInstead, our approach is to update the sample weights via the loss of the offline dataset in Eq. (7), a process that is detailed in lines 170 to 181 of our manuscript:\n\n$$\n\\mathcal{L}(\\boldsymbol{\\theta}^*(\\boldsymbol{\\omega}))= \\arg\\min_{\\boldsymbol{\\omega}}\\frac{1}{N}\\sum_{i=1}^N (f_{\\boldsymbol{\\theta}^*(\\boldsymbol{\\omega})}(\\boldsymbol{x}_i) - y_i)^2.\n$$\n\nIn line 180, we highlight that our meta-learning optimization strategy is designed to allocate higher weights to samples that exhibit similar gradients to those found in the offline dataset. This methodology helps us to avoid convergence to trivial solutions, thus enhancing the robustness and effectiveness of our approach.\n\n\n## Questions\n> In co-teaching, the two models other than proxy share knowledge about top-K points which both have learnt very well. \n\nThank you for your interesting observation. There are three key points to address your comment:\n\n1. The top-K points are more likely to be out-of-distribution as they are sampled along the gradient optimization path, aiming for higher-scoring designs than the offline dataset.\n\n2. Your point about the top-K points being well learned by all proxies, if they are in-distribution, is insightful. However, it's important to consider that our proxies are initialized randomly and, even though they are pretrained on the same offline dataset, their learning trajectories  differ. Consequently, each proxy grasps different knowledge, providing complementary perspectives, much like ensemble learning.\n\n3. As for focusing on the worst-K points, these typically correspond to samples where the two proxies largely disagree. Generally, these high-loss samples contain misleading information and directing the third proxy to fine-tune based on these samples could degrade its performance as we demonstrate on TF8 and Dkitty benchmarks in the table below.\n\n| Method  | Best-K Loss (Ours) | Worst-K Loss   |\n|---------|--------------------|----------------|\n| TF8     | **0.958 ± 0.008**  | 0.861 ± 0.058  |\n| D'Kitty | **0.968 ± 0.020**  | 0.958 ± 0.009  |\n\nThus, we believe our current approach of sharing knowledge about the top-K points is justified and leads to more robust learning dynamics.\n\n\n> I per-sample weighting\n\nThank you for suggesting the RGD method as a comparable solution in per-sample weighting. Though this work was published after our submission deadline, we have conducted additional experiments on the TFB8 and Dkitty benchmarks:\n\n| Method  | Sample Reweighting (Ours) | RGD-EXP        |\n|---------|---------------------------|----------------|\n| TF8     | **0.958 ± 0.008**         | 0.906 ± 0.058  |\n| D'Kitty | **0.968 ± 0.020**         | 0.955 ± 0.014  |\n\nAlthough RGD is efficient and theoretically effective, our results were slightly better, possibly due to our method's ability to utilize information from the static dataset. \n\nTo address this important work and offer a comprehensive perspective, we will update our manuscript to include a discussion of RGD. At line 313, we will add: \"Inspired by distributionally robust optimization, recent work [2] proposes a re-weighted gradient descent algorithm that provides an efficient and effective means of reweighting.\" Additionally, we will include the comparative experimental results in the Appendix and add a reference at Line 271: \"Our reweighting module is also compared with the recently proposed RGD method [2] as detailed in the Appendix.\"\n\n    [1] Shu, Jun, et al. \"Meta-weight-net: Learning an explicit mapping for sample weighting.\" Advances in neural information processing systems 32 (2019).\n    [2] Kumar, Ramnath, et al. \"Stochastic Re-weighted Gradient Descent via Distributionally Robust Optimization.\" arXiv preprint arXiv:2306.09222 (2023)."}, {"review_id": "iCX7PyXp31", "reviewer": "Reviewer_P8Ru", "summary": "The paper addresses the offline MBO problem, which aims to synthesize an instance $x$ that maximizes the proxy $f(\\cdot)$ estimated by a neural network. The popular approach, gradient ascent, updates the instance $x_{t+1}$ to have a higher proxy value compared to the previous instance $x_t$. However, this approach encounters a challenge where the proxy $f(\\cdot)$ becomes less accurate as the iterations progress, as the instance $x_t$ becomes more out-of-distribution.\n\nTo overcome this challenge, the paper proposes two techniques inspired by the field of noisy labels. Firstly, the paper applies co-teaching, a method that filters confident samples by selecting those consistent across two models. In this case, since the labels are generated by a neural network, the final method involves three models: one pseudo-labeler and two co-teachers. The three models are trained alternately, changing their roles. Secondly, to further enhance performance, the paper introduces sample reweighing, a technique that assigns higher weights to confident samples using meta-learning. When combined, the proposed ICT (importance-aware co-teaching) method outperforms previous approaches.", "questions": "Written in the weakness.", "limitations": "Discussed.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The idea of applying noisy label techniques to handle pseudo-labels of OOD samples makes sense.\n- The proposed two components are necessary, as shown in the ablation study.", "weaknesses": "### Experiments are not convincing\nMy biggest concern is the experimental results. The gains over the prior works are marginal compared to the high variances. The rank seems better at first glance, but it is not really convincing as the gap of proxy scores is too incremental.\n\n### On the evaluation metric\nI know this is not the problem of this paper but of the entire field. Still, reporting only the proxy value as the evaluation metric may not be enough. How can one confirm that the updated instance $x_T$ follows the valid form of $\\mathcal{X}$?\n\n### Limited technical novelty\nThe proposed method is basically an application of known methods such as co-teaching and sample reweighing. Although the three model parts were somewhat new, thanks to the use of the pseudo-labeler, the technical innovation is not sufficiently strong.\n\n### Presentation can be polished\nThe writing contains various repetitions, with similar words appearing in the abstract, introduction, method, and conclusion. It would be great if the paper could provide more information and prune redundant details. Additionally, Figure 1 and 2 are hard to understand without reading the full texts. The captions can be polished to be self-contained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## General Reply\nWe really appreciate your careful reviews and valuable comments. We provide responses to each comment below and explain how we address them in the paper.\n\n## Weakness\n > Experiments are not convincing… gains are marginal compared to the high variances\n\nIn order to better quantify the relative gains shown in our results, we have run a statistical significance test (Welch's t-test) between our method and the second-best method. The p-values are 0.437 on SuperC, 0.004 on Ant, 0.009 on D'Kitty, 0.014 on Hopper, 0.000 on TF8, 0.045 on TF10, 0.490 on NAS. This demonstrates our method has statistically significant improvement over other methods in a majority of tasks (5/7), further corroborating the assertions made in our manuscript. Furthermore, it is worth noting that the gains in previous works such as COMs, ROMA, NEMO, and BDI, are also often incremental. In this context, our contributions can be seen as significant, aligning with the challenging nature of the problem and the existing state of the art.\n\nIn Line 257, we will add: \" We have further run a Welch's t-test between our method and the second-best method, obtaining p-values of 0.437 on SuperC, 0.004 on Ant, 0.009 on D'Kitty, 0.014 on Hopper, 0.000 on TF8, 0.045 on TF10, 0.490 on NAS. This demonstrates statistically significant improvement in 5 out of 7 tasks, reaffirming the effectiveness of our method.\"\n\n > On the evaluation metric\n\nYour concern about relying solely on proxy values for evaluation is well-taken. In our original manuscript, we have addressed this by using Design-Bench [1] to provide direct oracle evaluation in six of our seven tasks (excluding Superconductor), thereby avoiding proxy evaluation. \n\nFor continuous tasks like Ant, D’Kitty, and Hopper, we use simulations for design scores. TFBind8 and TFBind10 tasks employ dictionary lookups, and the NAS task utilizes CIFAR10 training. No proxies are used for evaluation.\n\nFurthermore, we ensure the validity of our design instances by employing the encoding-decoding scheme outlined in [1]. This scheme ensures that our decoded designs - be it DNA sequences or network architectures or other designs - adhere to the valid form of $\\mathcal{X}$. \n\nExcluding the Superconductor benchmark, which uses proxy evaluation, our method still performs well in terms of ranking.\n\n| Method                         | Rank Mean         | Rank Median       |\n|-------------|-----------|----------|\n| BO-qEI                         |  9.0/15           |  10.0/15          |\n| CMA-ES                         |  5.0/15           |  2.5/15           |\n| REINFORCE                      |  11.3/15          |  15.0/15          |\n| CbAS                           |  9.8/15           |  10.0/15          |\n| Auto.CbAS                      |  11.2/15          |  11.0/15          |\n| MIN                            |  11.5/15          |  12.0/15          |\n| Grad                           |  7.3/15           |  7.0/15           |\n| Mean                           |  6.2/15           |  5.0/15           |\n| Min                            |  7.8/15           |  8.0/15           |\n| COMs                           |  10.0/15          |  10.5/15          |\n| ROMA                           |  5.5/15           |  6.0/15           |\n| NEMO                           |  4.6/15           |  4.0/15           |\n| BDI                            |  8.8/15           |  8.5/15           |\n| IOM                            |  8.7/15           |  7.5/15           |\n| ICT **(ours)**                 |  2.8/15           |  1.5/15           |\n \n > Limited technical novelty\n\nThank you for your observations on technical novelty. We acknowledge our work's basis in co-teaching and sample reweighing, but emphasize that the novelty lies in adapting these methods to the unique context of offline MBO. This includes:\n\n1. Management of Noise: Unlike traditional applications focused on label noise in existing datasets, our method employs a pseudo-labeler to dynamically generate data around the current optimization point. This new approach enables us to utilize co-teaching and sample reweighing to enhance proxy robustness along the gradient optimization path, different from their typical use.\n\n2. Application to Regression vs Classification: Although co-teaching and sample reweighing often target classification tasks, our work creatively leverages these techniques for a regression task within offline MBO by leveraging the task characteristics.\n\n3. Use of the Offline Dataset: In the domain of meta-learning-based sample reweighing, previous work often necessitates a clean validation set for updating sample weights. Our work instead utilizes an offline dataset for this purpose, further demonstrating novelty in our adaptation.\n\nOur work's novelty, thus, lies in the inventive adaptation of existing techniques to new challenges, enhancing offline MBO performance. The pseudo-labeler's role in this process underscores our method's unique contributions.\n\n > Presentation can be polished\n\nThank you for your constructive feedback on the presentation of our paper.\n\nWe recognize the redundancies and will work to eliminate them in the abstract, introduction, method, and conclusion. \n\nWe appreciate your suggestions regarding Figures 1 and 2. While we believe that the necessary information for understanding these figures is provided in close proximity in the text, we understand that the captions themselves could be more informative. While the captions were concise to avoid repetition, we recognize they could be more informative. Recognizing the importance of self-contained captions, we will revise them to be more comprehensive in the final version of our paper.\n\n## Overall\nDoes this address your concerns? We value your feedback and look forward to further discussion. Thank you.\n\n    [1] Mackenzie, Brandon, et al. \"Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization.\" arXiv preprint arXiv:2103.08738 (2021)."}, {"review_id": "P4MaOnGsYN", "reviewer": "Reviewer_VPeC", "summary": "The paper proposes a new method to address the out-of-distribution issue of offline model-based optimization.\nThe author first generates a pseudo-labeled dataset and identifies valuable (important) data for proxy co-teaching.\nTo alleviate potential inaccuracies in pseudo-labels, a meta-learning framework is proposed to adjust the importance weights of samples.\nExperiments on both continuous and discrete tasks demonstrate the effectiveness of the proposed method.", "questions": "Is it possible to visualize the so-called valuable samples such that the intuitive interpretation can be obtained?", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The out-of-distribution issue in offline model-based optimization is a challenging problem.\nThe proposed method is well-motivated, technically sound, and interesting. \n\n2. Experiments and ablation studies on both continuous and discrete tasks demonstrate the effectiveness of the proposed method.\n\n3. The paper is well-written and easy to follow.", "weaknesses": "1. In Lines 150-151, the authors claim that \"small-loss samples typically contain valuable knowledge, making them ideal for enhancing proxy robustness\". However, in my opinion, these small-loss samples might be some samples easy to learn since different proxies generate similar outputs for them. These easy samples may not be the truly valuable samples that benefit model generalization.\n\n2. In Section 3.2, a meta-learning framework is proposed to adjust the importance weights of samples. However, there still exist distributional shifts between the inner-loop and outer-loop. Therefore, the loss in Equation 7 may not be informative enough to identify truly important samples for generalization. Samples with hight weight might be the ones similar to the offline dataset. Therefore, the proposed method may not be applicable when the distributional shift is large.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## General Reply\nWe sincerely appreciate your comprehensive feedback and insights, which are very helpful to this paper. Below we respond back to each of your comments.\n\n## Weakness\n > In Lines 150-151, the authors claim that \"small-loss samples typically contain valuable knowledge, making them ideal for enhancing proxy robustness\". \n\nThank you for your insightful concern about small-loss samples. While it's true that these small-loss samples might be easy to learn and different proxies may generate similar outputs, in our noisy label context, these small-loss samples are more likely to be 'clean.' We have found that this makes them valuable for enhancing our model's robustness. \n\nThis idea finds support in the work of Meta-weight-net[1], where different approaches to identifying valuable samples are highlighted. In a clean data context, samples associated with large losses are indeed deemed more valuable as they present learning challenges. However, the scenario is reversed in a noisy data context, where small-loss samples are typically more reliable and valuable, as they are less likely to be influenced by noise.\n\nIn regards to the generation of these samples, our method follows a gradient optimization path. The gradient optimization not only gives us the current optimization point but also provides information about the samples around this point. Therefore, these samples may not strictly adhere to the training distribution which provides key insights into the local optimization landscape. Particularly, the small-loss samples in this context offer important knowledge that can effectively enhance the performance of our proxies around the current optimization point.\n\nYour remarks have helped us clarify these points, and we believe our approach significantly contributes to building more robust models in the presence of noisy labels.\n\n> In Section 3.2, a meta-learning framework is proposed to adjust the importance weights of samples. However, there still exist distributional shifts between the inner-loop and outer-loop. \n\nThank you for raising this important concern regarding the distributional shifts between the inner-loop and outer-loop. Indeed, this represents one of the key challenges in offline MBO, and we appreciate your insights.\n\nOur proposed method aims to mitigate the out-of-distribution issue but does not completely eliminate it. The offline dataset plays a crucial role in providing extrapolation guidance that is used to optimize the sample weights, particularly when the samples lie within reasonable proximity of the training distribution.\n\nThe samples that we generate along the gradient optimization path tend to move away from the original training distribution. These samples often provide valuable local information that is beneficial for learning. Considered 'near-the-boundary' instances, their weights can be strategically adjusted using the offline dataset, thereby aiding in training a more generalizable and robust proxy.\n\nHowever, when the samples are far removed from the training distribution, the extrapolation from the offline dataset for weight adjustments may not be as accurate. This problem of large distributional shifts is prevalent across the field of offline MBO and to our knowledge, no method has been able to solve it completely. Despite these challenges, our experiments have demonstrated the effectiveness of our method under a variety of conditions.\n\nYour constructive feedback will guide us in our future work to continue refining our method to better handle large distributional shifts.\n\n## Questions\n\n> Is it possible to visualize the so-called valuable samples such that the intuitive interpretation can be obtained?\n\nThanks for this suggestion. Visualizing these samples indeed poses an interesting challenge due to the high-dimensional nature of our data and the abstract concept of 'value' in this context. Rather than resorting to visual demonstrations, we've adopted an analytical approach to identify and validate the value of samples in our method. In our case, the 'value' of a sample is related to its pseudo-label accuracy, which might not directly translate into a visually intuitive metric. Therefore, we instead measure the value by comparing a sample's pseudo-label with its ground truth label, with a lower mean squared loss indicating a more valuable sample.\n\nWe conduct a thorough analysis in Appendix A.4 titled \"Analysis of Co-teaching and Sample Reweighting Efficacy\", focusing on the two key steps of our method: (1) pseudo-label-driven co-teaching, and (2) meta-learning-based sample reweighting. For both the D'Kitty (continuous) and TF8 (discrete) tasks, we demonstrate that our method successfully selects valuable samples that closely align with the ground truth.\n\nTo highlight, the selected samples under the co-teaching process and the samples assigned with larger weights in the reweighting step both have lower mean squared loss compared to their counterparts. These findings substantiate that our method effectively identifies valuable samples, although not visually, but in a way that we believe still presents a strong and rigorous validation of our approach.\n\n## Overall\n\nDoes this explanation address your concerns? Your feedback is useful to our paper, and we eagerly anticipate further dialogue during the rebuttal phase. Thank you.\n\n    [1] Shu, Jun, et al. \"Meta-weight-net: Learning an explicit mapping for sample weighting.\" Advances in neural information processing systems 32 (2019).\n"}, {"review_id": "df8FGrEolB", "reviewer": "Reviewer_DRKm", "summary": "This paper proposes a novel model-based optimization method that combines ensemble-based ideas and meta learning. The authors introduce three proxy models, each capable of generating pseudo-labels for the training of the other models, effectively making them co-teachers. This co-teaching process serves as an ensemble training method, mitigating potential risks associated with pseudo-labeling. Additionally, the paper incorporates meta learning techniques by re-weighting the samples, assigning higher weights to accurate samples. The intuitive nature of this idea and the reasonable performance observed in the experiments make a compelling case for accepting this paper.\n\n\n\n\n", "questions": "\n1. The paper employs three proxy co-teachers, but it would be interesting to investigate the impact of reducing the number to just two proxies. Are there any notable observations or differences in performance with this modification?\n\n2. The meta learning process appears to involve bi-level optimization of parameters and weights. Considering the potential complexity of this approach, it raises concerns about computational resources required to execute the code. It would be helpful to provide insights or strategies to address this issue.\n\n3. While this method demonstrates impressive performance in continuous design tasks, its effectiveness in discrete tasks, such as TFBind8, seems to be limited. How does this method perform on other protein tasks like GFP or UTR? Is there any intuition as to why this method excels specifically in continuous tasks?\n\n4. The method's performance, when measured by the median score (50th percentile), appears to be weaker in comparison to the maximum score (100th percentile). Can you provide some intuition or explanation for this observation? It would be valuable to understand the factors contributing to this difference in performance.", "limitations": "This paper really already gives limitations which I really appreciate it. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "\n1. There are several methods available for stabilizing the learning process with pseudo-labels, and this idea has the potential to be applied to various literature in the field.\n\n2. The idea presented in this paper is straightforward and effectively communicated, allowing readers to grasp its essence easily.\n\n3. One of the key contributions of this paper is the thorough analysis of hyperparameters, which play a critical role in model-based optimization. The authors provide valuable insights into optimizing these parameters to achieve better performance.\n", "weaknesses": "\n1. Although this paper presents a highly intuitive method, it lacks a detailed mathematical explanation for why this approach is effective. It would be beneficial to include analysis based on experiments to provide more insight into the working mechanism.\n\n2. It appears that several ideas in this paper draw inspiration from the prior method of BDI [1], particularly in the context of offline model-based optimization. While this viewpoint is intriguing, a more explicit analysis comparing the proposed method with BDI would greatly assist readers in understanding the similarities and differences.\n\n3. It would be valuable to reference offline biological design methods [2,3], even though they may not directly address continuous tasks. Specifically, [2] can be good reference to discuss this method due to their utilization of pseudo-labeling and their success in stabilizing the bootstrapping process, which aligns with the idea presented in this paper. Including a dedicated discussion section on these references would greatly benefit future researchers.\n\n[1] Chen, Can, et al. \"Bidirectional learning for offline infinite-width model-based optimization.\" Advances in Neural Information Processing Systems 35 (2022): 29454-29467.\n\n[2] Kim, Minsu, et al. \"Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequences.\" arXiv preprint arXiv:2306.03111 (2023).\n\n[3] Jain, Moksh, et al. \"Biological sequence design with gflownets.\" International Conference on Machine Learning. PMLR, 2022.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## General Reply\n\nWe appreciate your thoughtful comments. Please find our point-by-point responses to your feedback below.\n\n## Weakness\n> It lacks a detailed mathematical explanation\n\nThank you for your observation about the lack of detailed mathematical explanation. We agree that such analysis can deepen understanding, but in our context, theoretical assumptions required for mathematical modeling are difficult to formulate. Instead, we have chosen to focus on empirical analysis with experiments. Specifically, we've focused on thoroughly examining two key aspects of our method: (1) co-teaching and (2) sample reweighting, as outlined in Appendix A.4 of our paper.\n\nFor both D'Kitty and TF8, we demonstrated that the samples selected by co-teaching or assigned a larger weight achieve closer alignment with the ground truth. This suggests that our method indeed effectively identifies valuable samples.\n\n> Several ideas draw inspiration from BDI [1]\n\nWe agree with your observation that our work shares some similarities with BDI [1]. Like BDI, which utilizes backward mapping to optimize high-scoring designs through meta-learning, our method similarly uses meta-learning. Specifically, we use meta-learning to optimize sample weights of the pseudo-labeled data. Moreover, BDI and our method both incorporate sample weights to signify the relevance of high-scoring samples. Despite these similarities, our method has key differences from BDI, primarily in how sample weights are assigned and the unique application of co-teaching. \n\nTo clarify these, in Line 308, we'll add: 'Our work is related to BDI [1]. Like BDI, we utilize meta-learning and employ weights to highlight some samples. However, the crux of our approach lies in our unique co-teaching strategy which sets our method apart from BDI.'\n\n> It would be valuable to reference offline biological design methods [2,3]\n\nThank you for highlighting offline biological design methods [2,3]. They offer valuable insights, particularly [2], with its use of pseudo-labeling and success in stabilizing bootstrapping.\n\nIn Line 307, we will add: 'An interesting subfield of offline MBO includes biological sequence design, which has potential applications such as designing drugs for treating diseases [2,3]. In particular, [2] also adopts a proxy as a pseudo-labeler and aligns the generator with the proxy, a technique that resonates with our method.'\n\n## Questions\n\n> Interesting to two proxies. \n\nYour suggestion to explore using only two proxies is intriguing, but our method fundamentally requires three.\n\nIn our method, one proxy specifically serves as the pseudo-labeler. This proxy provides the pseudo-labeled dataset, upon which the other two proxies perform co-teaching. By reducing the proxies to two, we eliminate the possibility of co-teaching, which is a cornerstone of our method. Therefore, a minimum of three proxies is essential for our method.\n\n> complexity of bi-level optimization\n\nYou've correctly highlighted the potential complexity of bi-level optimization. Despite this, our design ensures efficiency. We've used strategies such as one-step approximation in Eq. (6) and avoided second-order gradient computations in Eq. (8), which significantly reduce computational demands. This efficient calculation is possible because the scalar sample weight is directly applied to the sample loss, which is not common in typical bi-level optimization. \n\nWe have compared computational time (seconds) for our ICT, gradient ascent, and ensemble methods on TF8 and D'Kitty, using a machine with an Intel i9-12900K CPU and an NVIDIA GeForce RTX 3090 GPU. The results, including time to train proxies and generate the top 128 designs, show comparable efficiency across methods:\n\n| Method  | ICT  | Grad   | Mean  |\n|----|-----|----|------|\n| TF8 | 170.4 | 51.1 | 146.3 |\n| D'Kitty | 294.5 | 231.2 | 271.6  |\n\n> its effectiveness in discrete tasks limited. \n\nWe acknowledge that our method shows impressive performance in continuous tasks but shows more-limited improvements in discrete ones like TFBind8. \n\nWe believe that the reduced performance in discrete tasks is largely due to the characteristics of the encoding-decoding strategy employed in the design-bench. The strategy does not take into account the sequential nature of biological sequences.\n\nOur method has shown promising results on other protein-related tasks, such as GFP(0.864 ± 0.000) and UTR(0.695 ± 0.034). These performances are comparable to the baselines in [1, 4]. We posit that these tasks, although they might seem discrete in nature, exhibit certain characteristics well-captured by our method.\n\n> The method's median score, appears to be weaker\n\nThe difference in our method's performance between the median score and maximum score arises from the nature of the evaluation. The median score reflects the middle value among the 128 generated candidates, where many comparison methods are proficient at identifying suitable designs, leading to similar median scores. However, the maximum score represents the optimal design, where methods must explore and exploit the design space far from the offline dataset.\n\nIn this context, our method distinguishes itself by effectively identifying the best possible designs. The median score provides a general comparison across methods, while the maximum score reveals our method's specific strength in pinpointing optimal solutions.\n\n## Overall\nDoes this address your concerns? We appreciate your feedback and look forward to further dialogue. Thank you.\n\n    [1] Chen, Can, et al. \"Bidirectional learning for offline infinite-width model-based optimization.\" NeurIPS 2022\n    [2] Kim, Minsu, et al. \"Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequences.\" 2023\n    [3] Jain, Moksh, et al. \"Biological sequence design with gflownets.\" ICML 2022.\n    [4] Brandon et al. Conservative objective models for effective offline model-based optimization. ICLR 2023."}]

Author rebuttal:

Meta-review: The paper introduces an Importance-aware Co-Teaching (ICT) approach for offline model-based optimization to tackle the out-of-distribution challenge. All reviewers find the idea of utilizing a pseudo-labeler to generate training data interesting and valuable. The effectiveness of the method is demonstrated through ablation studies conducted on both continuous and discrete tasks. The proposed solution for handling inaccuracies in pseudo-labeling using a meta-learning framework is reasonable. The paper is praised for its lucidity, significance in addressing out-of-distribution issues, and sufficient experimental validation. Overall, the meta-reviewer deems this paper's contribution worthy of publication. The authors are encouraged to incorporate the reviewers' feedback while preparing the final version.
Decision: Accept (poster)