Title: Importance-aware Co-teaching for Offline Model-based Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: offline model-based optimization

co-teaching

meta-learning

sample reweighting
Abstract: Offline model-based optimization aims to find a design that maximizes a property of interest using only an offline dataset, with applications in robot, protein, and molecule design, among others. A prevalent approach is gradient ascent, where a proxy model is trained on the offline dataset and then used to optimize the design. This method suffers from an out-of-distribution issue, where the proxy is not accurate for unseen designs. To mitigate this issue, we explore using a pseudo-labeler to generate valuable data for fine-tuning the proxy. 
Specifically, we propose $\textit{\textbf{I}mportance-aware \textbf{C}o-\textbf{T}eaching for Offline Model-based Optimization}~(\textbf{ICT})$. This method maintains three symmetric proxies with their mean ensemble as the final proxy, and comprises two steps. The first step is $\textit{pseudo-label-driven co-teaching}$. In this step, one proxy is iteratively selected as the pseudo-labeler for designs near the current optimization point, generating pseudo-labeled data.  Subsequently, a co-teaching process identifies small-loss samples as valuable data and exchanges them between the other two proxies for fine-tuning, promoting knowledge transfer.  This procedure is repeated three times, with a different proxy chosen as the pseudo-labeler each time, ultimately enhancing the ensemble performance.
To further improve accuracy of pseudo-labels, we perform a secondary step of $\textit{meta-learning-based sample reweighting}$,
which assigns importance weights to samples in the pseudo-labeled dataset and updates them via meta-learning. ICT achieves state-of-the-art results across multiple design-bench tasks, achieving the best mean rank $3.1$ and median rank $2$ among $15$ methods.
Our source code can be accessed here.

Importance-aware Co-teaching for Offline Model-based Optimization

Abstract

Offline model-based optimization aims to find a design that maximizes a property of interest using only an offline dataset, with applications in robot, protein, and molecule design, among others. A prevalent approach is gradient ascent, where a proxy model is trained on the offline dataset and then used to optimize the design. This method suffers from an out-of-distribution issue, where the proxy is not accurate for unseen designs. To mitigate this issue, we explore using a pseudolabeler to generate valuable data for fine-tuning the proxy. Specifically, we propose I mportance-aware C oT eaching for Offline Model-based Optimization ( ICT ). This method maintains three symmetric proxies with their mean ensemble as the final proxy, and comprises two steps. The first step is pseudo-label-driven co-teaching . In this step, one proxy is iteratively selected as the pseudo-labeler for designs near the current optimization point, generating pseudo-labeled data. Subsequently, a co-teaching process identifies small-loss samples as valuable data and exchanges them between the other two proxies for fine-tuning, promoting knowledge transfer. This procedure is repeated three times, with a different proxy chosen as the pseudolabeler each time, ultimately enhancing the ensemble performance. To further improve accuracy of pseudo-labels, we perform a secondary step of meta-learningbased sample reweighting , which assigns importance weights to samples in the pseudo-labeled dataset and updates them via meta-learning. ICT achieves state-ofthe-art results across multiple design-bench tasks, achieving the best mean rank of 3 . 1 and median rank of 2 , among 15 methods. Our source code can be found here.

1 Introduction

A primary goal in many domains is to design or create new objects with desired properties [1]. Examples include the design of robot morphologies [2], protein design, and molecule design [3, 4]. Numerous studies obtain new designs by iteratively querying an unknown objective function that maps a design to its corresponding property score. However, in real-world scenarios, evaluating the objective function can be expensive or risky [3-7]. As a result, it is often more practical to assume access only to an offline dataset of designs and their property scores. This type of problem is referred to as offline model-based optimization (MBO) [1]. The goal of MBO is to find a design that maximizes the unknown objective function using solely the offline dataset.

Gradient ascent is a common approach to address the offline MBO problem. For example, as illustrated in Figure 2 (a), the offline dataset may consist of three robot size and robot speed pairs p 1 , 2 , 3 . A simple DNN model, referred to as the vanilla proxy and represented as f θ ( · ) , is trained to fit the offline dataset as an approximation to the unknown objective function. Gradient ascent is subsequently applied to existing designs with respect to the vanilla proxy f θ ( · ) , aiming to generate a new design with a higher score. However, the gradient ascent method suffers from an out-ofdistribution issue, where the vanilla proxy cannot accurately estimate data outside of the training distribution, leading to a significant gap between the vanilla proxy and the ground-truth function, as shown in Figure 2 (a). As a consequence, the scores of new designs obtained via gradient ascent can be erroneously high [8, 9].

To mitigate the out-of-distribution issue, recent studies have suggested applying regularization techniques to either the proxy itself [8-10] or the design under consideration [11, 12]. These methods improve the proxy's robustness and generalization ability. However, a yet unexplored approach in this domain is using a pseudo-labeler to assign pseudo-labels to designs near the current point. Fine-tuning the proxy on this pseudo-labeled dataset can lead to improvement, provided that we can identify the valuable portion of the pseudo-labeled dataset.

Inspired by this, we propose I mportance-aware C oT eaching for Offline Model-based Optimization ( ICT ). This approach maintains three symmetric proxies, and their mean ensemble acts as the final proxy. ICT consists of two main steps with the first step being pseudo-label-driven co-teaching as illustrated in Figure 1. During this step, one proxy is iteratively selected as the pseudo-labeler, followed by a co-teaching process [13] that facilitates the exchange of valuable data between the other two proxies for fine-tuning. As depicted in Figure 1, there are three symmetric proxies, f θ 1 ( · ) , f θ 2 ( · ) , and f θ 3 ( · ) . The entire learning cycle (the larger triangle) can be divided into three symmetric parts (sub-triangles), with one proxy chosen to be the

f

Proxy

f

Pseudo-labeler

Co-teaching

LabelTransfer

$2

$3

f

f

f2

$3

$2

f3

pseudo-labeler in turn. Taking the top triangle as an example, we select f θ 1 ( · ) as the pseudo-labeler to generate pseudo labels for a set of points in the neighborhood of the current optimization point x t . The other two proxies, f θ 2 ( · ) and f θ 3 ( · ) , then receive the pseudo-labeled dataset. They compute the sample loss for each entry in the dataset and exchange small-loss samples between them for fine-tuning. This co-teaching process encourages knowledge transfer between the two proxies, as small losses are typically indicative of valuable knowledge. The symmetric nature of the three proxies allows the above process to repeat three times, with each proxyf θ 1 ( · ) , f θ 2 ( · ) , and f θ 3 ( · ) -taking turns as the pseudo-label generator. This learning cycle promotes the sharing of valuable knowledge among the three symmetric proxies, allowing them to collaboratively improve the ensemble performance in handling out-of-distribution designs.

(a)

(b)

1.50

Pseudo-labeled Dataset

1.50

War

from Co-teaching

1.25

OfflineDataset

Proxy f62

Fine-tuned

1.25

3

Speed

pb

Robot Speed

Pb

1.00

1.00

Robot

Pa

Pa

0.75

p3

Ground

0.75

P3

Ground

()f unnu1

Truth f(-)

p2

P2

Meta-learning

0.50

P1

0.50

P1

0.25

0.25

0.5

1.0

1.5

0.5

1.0

1.5

Despite the efforts made in the first step, small-loss data may still contain inaccurate labels. During the first step, small-loss data ( p a and p b ) from the pseudo-labeled dataset produced by f θ 1 ( · ) are identified based on the predictions of proxy f θ 3 ( · ) and fed to proxy f θ 2 ( · ) . However, as shown in Figure 2 (a), the less accurate point p b deviates noticeably from the ground-truth, causing the fine-tuned proxy f θ 2 ( · ) to diverge from the ground-truth function. To address this, we introduce the second step of ICT, meta-learning-based sample reweighting , which aims to assign higher weights to more accurate points like p a and lower weights to less accurate ones like p b . To accomplish this, we assign an importance weight for every sample yielded by the first step ( ω a for p a and ω b for p b ) and propose a meta-learning framework to update these sample weights ( ω a and ω b ) automatically by leveraging the supervision signals from the offline dataset p 1 , 2 , 3 . Specifically, the proxy fine-tuned on the weighted small-loss data ( p a and p b ) is expected to perform well on the offline dataset, provided the weights are accurate, i.e., large ω a and small ω b . We can optimize the sample weights by minimizing the loss on the offline dataset as a function of the sample weights. As illustrated in Figure 2 (b), the weight of p a is optimized to be high, while the weight of p b is optimized to be low. Consequently, the proxy f ( b ) θ 2 ( · ) fine-tuned on the weighted samples in Figure 2 (b) is brought closer to the ground-truth objective function f ( · ) , compared to the case where the fine-tuned proxy f ( a ) θ 2 ( · ) is far from f ( · ) in Figure 2 (a). Through extensive experiments across various tasks [1], ICT proves effective at mitigating out-of-distribution issues, delivering state-of-the-art results.

In summary, our paper presents three main contributions:

We introduce I mportance-aware C oT eaching ( ICT ) for offline MBO. ICT consists of two steps. In the pseudo-label-driven co-teaching step, a proxy is iteratively chosen as the pseudo-labeler, initiating a co-teaching process that facilitates knowledge exchange between the other two proxies.

The second step, meta-learning-based sample reweighting , is introduced to alleviate potential inaccuracies in pseudo-labels. In this step, pseudo-labeled samples are assigned importance weights, which are then optimized through meta-learning.

Extensive experiments demonstrate ICT's effectiveness in addressing out-of-distribution issues, yielding state-of-the-art results in multiple MBO tasks. Specifically, ICT secures the best mean rank of 3 . 1 and median rank of 2 , among 15 methods.

2 Preliminaries

Offline model-based optimization (MBO) targets a variety of optimization problems with the goal of maximizing an unknown objective function using an offline dataset. Consider the design space X = R d , where d represents the design dimension. Formally, the offline MBO can be expressed as:

where f ( · ) denotes the unknown objective function, and x ∈ X denotes a candidate design. In this scenario, an offline dataset D = { ( x i , y i ) } N i =1 is available, where x i represents a specific design, such as robot size, and y i represents the corresponding score, like robot speed. In addition to robot design, similar problems also include protein and molecule design.

A common strategy for tackling offline MBO involves approximating the unknown objective function f ( · ) using a proxy function, typically represented by a deep neural network (DNN) f θ ( · ) , which is trained on the offline dataset:

With the trained proxy, design optimization is performed using gradient ascent steps:

Here, T denotes the number of steps, and η signifies the learning rate. The optimal design x ∗ is acquired as x T . This gradient ascent approach is limited by an out-of-distribution issue , as the proxy f θ ( x ) may not accurately predict scores for unseen designs, leading to suboptimal solutions.

3 Method

In this section, we introduce I mportance-aware C oT eaching ( ICT ), which consists of two steps. We maintain three symmetric proxies and compute the mean ensemble as the final proxy. In Sec 3.1, we describe the first step, pseudo-label-driven co-teaching . This step involves iteratively selecting one proxy as the pseudo-label generator and implementing a co-teaching process to facilitate the exchange of valuable data between the remaining two proxies. Nevertheless, the samples exchanged during coteaching might still contain inaccurate labels, which necessitates the second step meta-learning-based sample reweighting in Sec 3.2. During this step, each sample from the previous step is assigned an importance weight and updated via meta-learning. Intuitively, the ICT process can be likened to an enhanced paper peer review procedure between three researchers preparing for submission. Each researcher, acting as an author, presents his/her paper to the other two. These two serve as reviewers and co-teach each other important points to better comprehend the paper, ultimately providing their feedback to the author. A detailed depiction of the entire algorithm can be found in Algorithm 1.

3.1 Pseudo-label-driven Co-teaching

Vanilla gradient ascent, as expressed in Eq. (3), is prone to out-of-distribution issues in offline model-based optimization. One potential yet unexplored solution is using a pseudo-labeler to provide pseudo-labels to designs around the optimization point. By fine-tuning the proxy using the valuable portion of the pseudo-labeled dataset, we can enhance the proxy's performance. To achieve this, we maintain three proxies simultaneously, computing their mean ensemble as the final proxy, and iteratively select one proxy to generate pseudo-labeled data. The other two proxies exchange knowledge estimated to have high value, by sharing small-loss data. Due to the symmetric nature of the three proxies, this process can be repeated three times for sharing valuable knowledge further.

Pseudo-label. We initially train three proxies f θ 1 ( · ) , f θ 2 ( · ) , and f θ 3 ( · ) on the whole offline dataset using Eq. (2) with different initializations, and conduct gradient ascent with their mean ensemble,

where η is the gradient ascent learning rate. Given the current optimization point x t , we sample M points x t, 1 , x t, 2 , . . . , x t,M around x t as x t,m = x t + γϵ , where γ is the noise coefficient and ϵ is drawn from the standard Gaussian distribution. An alternative way is to directly sample the M points around the offline dataset, rather than the current optimization point. We detail this option in Appendix A.1. We iteratively choose one proxy, for example f θ 1 ( · ) , to label these points, creating a pseudo-labeled dataset D 1 = { ( x t,j , f θ 1 ( x t,j )) } j M =1 . Lines 5 to 6 of Algorithm 1 detail the implementation of this segment.

Co-teaching. For each sample in the pseudo-labeled dataset D 1 , we compute the sample loss for f θ 2 ( · ) and f θ 3 ( · ) . Specifically, the losses are calculated as L 2 ,i = ( f θ 2 ( x t,i ) -f θ 1 ( x t,i )) 2 and L 3 ,i = ( f θ 3 ( x t,i ) -f θ 1 ( x t,i )) 2 , respectively. Small-loss samples typically contain valuable knowledge, making them ideal for enhancing proxy robustness [13]. Proxies f θ 2 ( · ) and f θ 3 ( · ) then exchange the top K small-loss samples as valuable data to teach each other where K is a hyperparameter. The co-teaching process enables the exchange of valuable knowledge between proxies f θ 2 ( · ) and f θ 3 ( · ) . This part is implemented as described in Lines 7 to 8 of Algorithm 1. The symmetric design of the three proxies, f θ 1 ( · ) , f θ 2 ( · ) , and f θ 3 ( · ) , enables the entire process to be iterated three times with one proxy chosen as the pseudo-labeler every time.

3.2 Meta-learning-based Sample Reweighting

While the previous step effectively selects samples for fine-tuning, these samples may still contain inaccuracies. To mitigate this, we introduce a meta-learning-based sample reweighting step. In this step, each sample obtained from the prior step is assigned an importance weight, which is then updated using a meta-learning framework. Without loss of generality, we use f θ ( · ) to represent any of f θ 1 ( · ) , f θ 2 ( · ) and f θ 3 ( · ) as this step applies identically to all three proxies. The top K small-loss samples selected from the previous step for fine-tuning f θ ( · ) are denoted as D s = { ( x s i , ¯ y s i ) } K i =1 .

Sample Reweighting. We assign an importance weight ω i to the i th selected sample and initialize these importance weights to ones. We expect smaller importance weights for less accurate samples

and larger importance weights for more accurate samples to improve proxy fine-tuning. With these weights, we can optimize the proxy parameters as follows:

Since we only want to perform fine-tuning based on D s , we can adopt one step of gradient descent:

where α is the learning rate for fine-tuning. This part is presented in Line 10 in Algorithm 1.

Meta-learning. The challenge now is finding a group of proper weights ω . We achieve this by leveraging the supervision signals from the offline dataset, which are generally accurate. If the sample weights are accurate, the proxy fine-tuned on the weighted samples is expected to perform well on the offline dataset. This is because the weighted samples aim to reflect the underlying ground-truth function that the offline dataset already captures, and both sets of data share common patterns. We can optimize the sample weights by minimizing the loss of the offline dataset in a meta-learning framework. The loss on the offline dataset can be written as:

The sample weight ω i for the i th sample can be updated by gradient descent:

where β is the learning rate for the meta-learning framework. From Eq. (8), it is worth mentioning that ∂ L ( θ ∗ ( ω )) ∂ θ ∂ ( f θ ( x s i ) -¯ y s i ) 2 ∂ θ ⊤ represents the similarity between the gradient of the offline dataset and the gradient of the i th sample. This implies that a sample with a gradient similar to the offline dataset will receive a higher weight and vice versa, revealing the inner mechanism of this framework. By applying the updated sample weights to Eq. (6) for fine-tuning, we improve the proxy's performance. This process is iteratively applied to each proxy, yielding a stronger ensemble. Lines 11 to 13 of Algorithm 1 showcase the execution of this part.

4 Experimental Results

4.1 Dataset and Evaluation

Dataset and Tasks. In this study, we conduct experiments on four continuous tasks and three discrete tasks. The continuous tasks include: (a) Superconductor (SuperC)[5], where the objective is to develop a superconductor with 86 continuous components to maximize critical temperature, using 17 , 010 designs; (b) Ant Morphology (Ant)[1, 14], where the aim is to design a quadrupedal ant with 60 continuous components to improve crawling speed, based on 10 , 004 designs; (c) D'Kitty Morphology (D'Kitty)[1, 15], where the focus is on shaping a quadrupedal D'Kitty with 56 continuous components to enhance crawling speed, using 10 , 004 designs; (d) Hopper Controller (Hopper)[1], where the aim is to identify a neural network policy with 5 , 126 weights to optimize return, using 3 , 200 designs. Additionally, our discrete tasks include: (e) TF Bind 8 (TF 8 )[6], where the goal is to discover an 8 -unit DNA sequence that maximizes binding activity score, utilizing 32 , 898 designs; (f) TF Bind 10 (TF 10 )[6], where the aim is to find a 10 -unit DNA sequence that optimizes binding activity score, using 50 , 000 designs; (g) NAS [16], where the objective is to find the optimal neural network architecture to enhance test accuracy on the CIFAR10 [17] dataset, using 1 , 771 designs.

Evaluation and Metrics. In accordance with the evaluation protocol used in [1, 11], we identify the top 128 designs from the offline dataset for each approach and report the 100 th percentile normalized

Algorithm 1 Importance-aware Co-teaching

ground-truth score. This score is computed as y n = y -y min y max -y min , where y min and y max represent the minimum and maximum scores within the entire unobserved dataset, respectively. The 50 th percentile (median) normalized ground-truth scores are included in Appendix A.2. For a better comparison, we report the best design in the offline dataset, denoted as D ( best ). We also provide mean and median rankings across all seven tasks for a broad performance assessment.

4.2 Comparison Methods

We compare our approach with two categories of baselines: ( 1 ) those that use generative models for sampling purposes, and ( 2 ) those that apply gradient updates derived from existing designs. The generative model-based methods learn and sample from the distribution of high-scoring designs, including: (i) MIN [18], which maps scores to designs and searches this map for optimal designs; (ii) CbAS [19], which uses a VAE model to adapt the design distribution towards high-scoring areas; (iii) Auto.CbAS [20], which employs importance sampling to retrain a regression model based on CbAS.

The second category encompasses: (i) Grad: carries out a basic gradient ascent on existing designs to generate new ones; (ii) Grad. Min: optimizes the lowest prediction from an ensemble of learned objective functions; (iii) Grad. Mean: optimizes the ensemble's mean prediction; (iv) ROMA [8]: applies smoothness regularization on the DNN; (v) COMs [9]: uses regularization to assign lower scores to designs obtained through gradient ascent; (vi) NEMO [10]: constrains the gap between the proxy and the ground-truth function via normalized maximum likelihood before performing gradient ascent; (vii) BDI [11] uses forward and backward mappings to distill knowledge from the offline dataset to the design; (viii) IOM [21]: enforces representation invariance between the training dataset and the optimized designs.

We also compare with traditional methods in [1]: (i) CMA-ES [22]: gradually adjusts the distribution towards the optimal design by modifying the covariance matrix. (ii) BO-qEI [23]: executes Bayesian Optimization to maximize the proxy, suggests designs through the quasi-Expected-Improvement acquisition function, and labels the designs using the proxy function. (iii) REINFORCE [24]: optimizes the distribution over the input space using the learned proxy.

4.3 Training Details

We adopt the training settings from [1] for all comparison methods unless otherwise specified. We use a 3 -layer MLP (MultiLayer Perceptron) with ReLU activation for all gradient updating methods, and set the hidden size to 2048 . Additional hyperparameter details are elaborated in Appendix A.3.

One of the top 128 designs from the offline dataset is iteratively selected as the starting point, as outlined in Line 2 of Algorithm 1. We reference results from [1] for non-gradient-ascent methods such as BO-qEI, CMA-ES, REINFORCE, CbAS, and Auto.CbAS. For gradient-based methods, we run each setting over 8 trials and report the mean and standard error. All experiments are run on a single NVIDIA GeForce RTX 3090 GPU.

4.4 Results and Analysis

Performance in Continuous Tasks. Table 1 presents the results across different continuous domains. In all four continuous tasks, our ICT method achieves the top performance. Notably, it surpasses the basic gradient ascent, Grad, demonstrating its ability to mitigate the out-of-distribution issue. The superior performance of Grad.mean over Grad can be attributed to the ensemble model's robustness in making predictions [25]. Furthermore, ICT generally outperforms ensemble methods and other gradient-based techniques such as COMs and ROMA, demonstrating the effectiveness of our strategy. Generative model-based methods, such as CbAS and MINs, however, struggle with the high-dimensional task Hopper Controller. Interestingly, ICT necessitates only three standard proxies and avoids the need for training a generative model, which can often be a challenging task. These results indicate that ICT is a simple yet potent baseline for offline MBO.

Table 2: Experimental results on discrete tasks, and ranking on all tasks for comparison.

Performance in Discrete Tasks. Table 2 showcases the outcomes across various discrete domains. ICT attains top performances in two out of the three tasks, TF Bind 8 and TF Bind 10 . These results suggest that ICT is a powerful method in the discrete domain. However, in NAS, the performance of ICT is not as strong, which can be attributed to two factors. Firstly, the neural network design in NAS,

represented by a 64 -length sequence of 5 -categorical one-hot vectors, has a higher dimensionality than TF Bind 8 and TF Bind 10 , making the optimization process more complex. Furthermore, the simplistic encoding-decoding strategy in design-bench may not accurately capture the intricacies of the neural network's accuracy, which can only be determined after training on CIFAR10.

Summary. ICT attains the highest rankings with a mean of 3 . 1 / 15 and median of 2 / 15 as shown in Table 2 and Figure 3, and also secures top performances in 6 out of the 7 tasks. We have further run a Welch's t-test between our method and the second-best method, obtaining p-values of 0 . 437 on SuperC, 0 . 004 on Ant, 0 . 009 on D'Kitty, 0 . 014 on Hopper, 0 . 000 on TF8, 0 . 045 on TF10, 0 . 490 on NAS. This demonstrates statistically significant improvement in 5 out of 7 tasks, reaffirming the effectiveness of our method.

4.5 Ablation Studies

To better understand the impact of pseudo-label-driven coteaching (co-teaching) and metalearning-based sample reweighting (reweighting) on the performance of our proposed ICT method, we conduct ablation studies by removing either co-teaching or reweighting from the full ICT approach. Table 3 presents the results. Beyond

just assessing these performance indicators, we also verify the accuracy of the samples chosen by co-teaching, as well as the efficacy of the sample weights we have calculated. We do this by referring to the ground truth, with further details provided in Appendix A.4. Our reweighting module is also compared with the recently proposed RGD method [26] as detailed in the Appendix A.5.

For two of the discrete tasks (TF 8 and TF 10 ), the ICT method consistently exceeds the performance of both its ablated versions. This highlights the efficacy of the two steps when handling discrete tasks. Conversely, the exclusion of the co-teaching in NAS leads to an increase in performance. This could be attributed to the fact that the encoding-decoding strategy of NAS in design-bench is unable to accurately capture the inherent complexity of neural networks. As such, the co-teaching step, reliant on this strategy, might not be as effective. For the continuous tasks (SuperC, Ant, D'Kitty, and Hopper), we observe that the complete ICT method consistently achieves superior performance. This underlines the effectiveness of the two steps when dealing with continuous tasks. The performance gains are particularly substantial in the Hopper task when the complete ICT method is compared with the ablated versions, illustrating the power of the two steps in managing high-dimensional continuous tasks. Overall, our ablation studies demonstrate that the inclusion of both co-teaching and reweighting in the ICT method generally enhances performance across diverse tasks and input dimensions, underscoring their integral role in our approach.

4.6 Hyperparameter Sensitivity

We first assess the robustness of our ICT method by varying the number of samples ( K ) selected during the co-teaching process on the continuous D'Kitty Morphology task. For this analysis, K is varied among K = 8 , 16 , 32 , 64 . In Figure 4 (a), we illustrate the 100 th percentile normalized ground-truth score as a function of time step T , for each of these K values. The results demonstrate that the performance of ICT is resilient to variations in K , maintaining performances within a certain range. Additionally, ICT is capable of generating high-scoring designs early on in the process, specifically achieving such designs around the time step t = 50 , and sustains this performance thereafter, demonstrating its robustness against the number of optimization steps T .

We further evaluate the robustness of our ICT method against the learning rate ( β ) for the metalearning framework. As depicted in Figure 4 (b), ICT's performance remains relatively consistent across a variety of β values, further demonstrating ICT's robustness with respect to the hyperparameter β . We explore the fine-tuning learning rate α and conduct further experiments and analysis on TF Bind 8. Details can be found in Appendix A.6.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15

Rank

BO-qEI

CMA-ES

REINFORCE

CbAS

Auto CbAS

MIN

Grad

Mean

Min

COMs

ROMA

NEMO

BDI

IOM

ICT (Ours)

(a)

(b)

0.95

0.95

Score

Score

0.90

0.90

S

S

β=4e-1

0.85

K=32

0.85

β=2e-1

β=1e-1

K=16

β=8e-2

1

β=5e-2

0.80

0

50

Step T

100

150

200

0.80

0

50

StepT

100

150

200

5 Related Works

Offline Model-based Optimization. Contemporary offline model-based optimization methods can be generally classified into two primary groups: (i) generating novel designs through generative models, and (ii) conducting gradient ascent on existing designs. The former methods learn and sample from the distribution of high-scoring designs including MIN [18], CbAS [19], Auto.CbAS [20] and BootGen [27]. Recently, gradient-based methods have gained popularity due to their ability to leverage deep neural networks (DNNs) for improved design generation. These methods apply regularization techniques to either the proxy itself [8-10] or the design under consideration [11, 12], enhancing the proxy's robustness and generalization capabilities. An interesting subfield of offline MBOincludes biological sequence design, which has potential applications such as designing drugs for treating diseases [27, 28]. In particular, the work [27] also adopts a proxy as a pseudo-labeler and aligns the generator with the proxy, a technique that resonates with our method. ICT falls under this category, but adopts a unique approach to improve proxy performance: it incorporates valuable knowledge from a pseudo-labeled dataset into other proxies for fine-tuning, thereby enhancing the ensemble performance. Notably, while the concurrent work of parallel mentoring [29] also employs pseudo-labeling, it focuses on pairwise comparison labels, potentially sacrificing some information due to its discrete nature.

Sample Reweighting. Sample reweighting is commonly utilized to address the issue of label noise [30, 31], where each sample is assigned a larger weight if it is more likely to be accurate, using a carefully designed function. Recent studies [32-34] suggest using a meta-set to guide the learning of sample weights, which can enhance model training. Such an approach is grounded in a meta-learning framework which can be used to learn hyperparameters [35-37, 34, 38-43]. Inspired by distributionally robust optimization, recent work [26] proposes a re-weighted gradient descent algorithm that provides an efficient and effective means of reweighting. In this paper, the pseudolabeled dataset generated by co-teaching may still contain some inaccuracies, while the offline dataset is generally accurate. We propose a sample reweighting framework to reduce the inaccuracies in the pseudo-labeled dataset by leveraging the supervision signals from the offline dataset.

Co-teaching. Co-teaching [13] is an effective technique for mitigating label noise by leveraging insights from peer networks. It involves the concurrent training of two proxies where one proxy identifies small-loss samples within a noisy mini-batch for fine-tuning the other. Co-teaching bears similarities to decoupling [44] and co-training [45], as they all involve the interaction between two models to enhance the training process. In this study, we adapt co-teaching to work with a pseudo-labeled dataset generated by a trained proxy, instead of relying on a noisy original dataset. Specifically, we employ one proxy to select accurate samples from this pseudo-labeled dataset for fine-tuning the other, and vice versa.

6 Conclusion and Discussion

In this study, we introduce the ICT (Importance-aware Co-Teaching) method for mitigating the out-of-distribution issue prevalent in offline model-based optimization. ICT is a two-step approach. The first step is pseudo-label-driven co-teaching, which iteratively selects a proxy to generate pseudolabeled data. Valuable data are identified by co-teaching to fine-tune other proxies. This process,

repeated three times with different pseudo-labelers, facilitates knowledge transfer. In the second step, meta-learning-based sample reweighting assigns and updates importance weights to samples selected by the co-teaching process, further improving the proxy fine-tuning. Our experimental findings demonstrate the success of ICT. We discuss its limitations in Appendix A.7

Future Work. Though we initially design ICT with three proxies, the method's inherent scalability and flexibility make it applicable to scenarios involving N proxies. In such a scenario, we can iteratively select one proxy out of N as the pseudo-labeler to generate data. Then, each of the remaining N -1 proxies could select small-loss samples from its perspective and provide these samples to the other N -2 proxies for fine-tuning. This process enhances knowledge transfer and facilitates cooperative learning among the proxies. Looking to the future, we plan to conduct further research into the dynamics of such an expanded ensemble of proxies.

Negative Impact. It is crucial to recognize that ICT's potential benefits come with possible negative consequences. Advanced optimization techniques can be applied for both constructive and destructive purposes, depending on their use. For example, while drug development and material design can have a positive impact on society, these techniques could also be misused to create harmful substances or products. As researchers, we must remain attentive and strive to ensure that our work is employed for the betterment of society while addressing any potential risks and ethical concerns.

References

Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-bench: Benchmarks for data-driven offline model-based optimization. arXiv preprint arXiv:2202.08450 , 2022.

Thomas Liao, Grant Wang, Brian Yang, Rene Lee, Kristofer Pister, Sergey Levine, and Roberto Calandra. Data-efficient learning of morphology and controller for a microrobot. arXiv preprint arXiv:1905.01334 , 2019.

Karen S Sarkisyan et al. Local fitness landscape of the green fluorescent protein. Nature , 2016.

Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. In Proc. Int. Conf. Learning Rep. (ICLR) , 2019.

Kam Hamidieh. A data-driven statistical model for predicting the critical temperature of a superconductor. Computational Materials Science , 2018.

Luis A Barrera et al. Survey of variation in human transcription factors reveals prevalent dna binding changes. Science , 2016.

Paul J Sample, Ban Wang, David W Reid, Vlad Presnyak, Iain J McFadyen, David R Morris, and Georg Seelig. Human 5 UTR design and variant effect prediction from a massively parallel translation assay. Nature Biotechnology , 2019.

Sihyun Yu, Sungsoo Ahn, Le Song, and Jinwoo Shin. Roma: Robust model adaptation for offline model-based optimization. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2021.

Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Conservative objective models for effective offline model-based optimization. In Proc. Int. Conf. Learning Rep. (ICLR) , 2021.

Justin Fu and Sergey Levine. Offline model-based optimization via normalized maximum likelihood estimation. Proc. Int. Conf. Learning Rep. (ICLR) , 2021.

Can Chen, Yingxue Zhang, Jie Fu, Xue Liu, and Mark Coates. Bidirectional learning for offline infinite-width model-based optimization. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2022.

Can Chen, Yingxue Zhang, Xue Liu, and Mark Coates. Bidirectional learning for offline model-based biological sequence design. In Proc. Int. Conf. Machine Lea. (ICML) , 2023.

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2018.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. Robel: Robotics benchmarks for learning with low-cost robots. In Conf. on Robot Lea. (CoRL) , 2020.

Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 , 2017.

Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580 , 2012.

Aviral Kumar and Sergey Levine. Model inversion networks for model-based optimization. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2020.

David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. In Proc. Int. Conf. Machine Lea. (ICML) , 2019.

Clara Fannjiang and Jennifer Listgarten. Autofocused oracles for model-based design. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2020.

Han Qi, Yi Su, Aviral Kumar, and Sergey Levine. Data-driven model-based optimization via invariant representation learning. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2022.

Nikolaus Hansen. The CMA evolution strategy: A comparing review. In Towards a New Evolutionary Computation: Advances in the Estimation of Distribution Algorithms , 2006.

James T Wilson, Riccardo Moriconi, Frank Hutter, and Marc Peter Deisenroth. The reparameterization trick for acquisition functions. arXiv preprint arXiv:1712.00424 , 2017.

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning , 1992.

Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2019.

Ramnath Kumar, Kushal Majmundar, Dheeraj Nagaraj, and Arun Sai Suggala. Stochastic re-weighted gradient descent via distributionally robust optimization. arXiv preprint arXiv:2306.09222 , 2023.

Minsu Kim, Federico Berto, Sungsoo Ahn, and Jinkyoo Park. Bootstrapped training of score-conditioned generator for offline design of biological sequences. arXiv preprint arXiv:2306.03111 , 2023.

Moksh Jain, Emmanuel Bengio, Alex Hernandez-Garcia, Jarrid Rector-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In Proc. Int. Conf. Machine Lea. (ICML) . PMLR, 2022.

Can Chen, Christopher Beckham, Zixuan Liu, Xue Liu, and Christopher Pal. Parallel-mentoring for offline model-based optimization. In Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2023.

Lu Jiang, Deyu Meng, Teruko Mitamura, and Alexander G Hauptmann. Easy samples first: Self-paced reranking for zero-example multimedia search. In Proceedings of the 22nd ACM international conference on Multimedia , 2014.

Yixin Wang, Alp Kucukelbir, and David M Blei. Robust probabilistic modeling with bayesian data reweighting. In Proc. Int. Conf. Machine Lea. (ICML) , 2017.

Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In Proc. Int. Conf. Machine Lea. (ICML) , 2018.

Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Metaweight-net: Learning an explicit mapping for sample weighting. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2019.

Can Chen, Shuhao Zheng, Xi Chen, Erqun Dong, Xue Steve Liu, Hao Liu, and Dejing Dou. Generalized data weighting via class-level gradient manipulation. Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2021.

Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In Proc. Int. Conf. Machine Lea. (ICML) , 2018.

Can Chen, Xi Chen, Chen Ma, Zixuan Liu, and Xue Liu. Gradient-based bi-level optimization for deep learning: A survey. arXiv preprint arXiv:2207.11719 , 2022.

Fuyuan Lyu, Xing Tang, Huifeng Guo, Ruiming Tang, Xiuqiang He, Rui Zhang, and Xue Liu. Memorize, factorize, or be naive: Learning optimal feature interaction methods for ctr prediction. In 2022 IEEE 38th International Conference on Data Engineering (ICDE) , 2022.

Can Chen, Chen Ma, Xi Chen, Sirui Song, Hao Liu, and Xue Liu. Unbiased implicit feedback via bi-level optimization. arXiv preprint arXiv:2206.00147 , 2022.

Can Chen, Jingbo Zhou, Fan Wang, Xue Liu, and Dejing Dou. Structure-aware protein selfsupervised learning. Bioinformatics , 2023.

Tommaso Giovannelli, Griffin Kent, and Luis Nunes Vicente. Bilevel optimization with a multi-objective lower-level problem: Risk-neutral and risk-averse formulations. arXiv preprint arXiv:2302.05540 , 2023.

Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama. Dataset distillation using parameter pruning. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences , 2023.

Zhixiang Chi, Yang Wang, Yuanhao Yu, and Jin Tang. Test-time fast adaptation for dynamic scene deblurring via meta-auxiliary learning. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recogn. (CVPR) , 2021.

Zhixiang Chi, Li Gu, Huan Liu, Yang Wang, Yuanhao Yu, and Jin Tang. Metafscil: A metalearning approach for few-shot class incremental learning. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recogn. (CVPR) , 2022.

Eran Malach and Shai Shalev-Shwartz. Decoupling" when to update" from" how to update". Proc. Adv. Neur. Inf. Proc. Syst (NeurIPS) , 2017.

Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the eleventh annual conference on Computational learning theory , 1998.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. Int. Conf. Learning Rep. (ICLR) , 2015.

A Appendix

A.1 Considerations for Sampling Around the Offline Dataset

In this subsection, we explore an alternative sampling strategy for the pseudo-labeling process. Instead of generating new samples around the current optimization point, this strategy generates samples directly around the offline dataset D . To ascertain the effectiveness of our chosen strategy against this alternative, we perform experiments on two tasks: D'Kitty (continuous) and TF 8 (discrete).

Table 4 showcases the results. For both tasks, our strategy consistently yields higher scores, affirming its superior performance over the alternative. The advantage of our chosen strategy can be attributed to its dynamic nature. By sampling around the current optimization point, we gather more insightful information for the local fine-tuning of the proxy. This strategy allows the co-teaching process to adapt and evolve according to the optimization trajectory, leading to improved performances.

A.2 Comparative Performance Analysis using Median Scores

In addition to the maximum scores discussed in the main paper, we also present the median ( 50 th percentile) scores across all seven tasks. The best design in the offline dataset, denoted as D ( best ), along with the mean and median rankings are provided for comprehensive comparison.

Performance in Continuous Tasks. Table 5 illustrates the performances of ICT compared with other methods in continuous tasks. It is noteworthy that ICT exhibits performance on par with the bestperforming methods. Compared with the vanilla gradient ascent (Grad), ICT demonstrates superior performance, thus affirming its effectiveness in addressing out-of-distribution issues. Moreover, ICT is generally better than the mean ensemble (Mean), which demonstrates the effectiveness of our strategy. These results support the use of ICT as a robust baseline for offline MBO.

Performance in Discrete Tasks. The median scores for discrete tasks are reported in Table 6. ICT consistently demonstrates high performance for both TF Bind 8 and TF Bind 10 . However, for the NAS task, which has a higher dimensionality than the two tasks, the optimization process becomes notably more complex. Further, the simplistic encoding-decoding strategy employed in the design bench may not accurately capture the intricacies of the neural network's accuracy, potentially contributing to ICT's suboptimal performance on the NAS task.

Summary. ICT excels by achieving the best median ranking and a top-two mean ranking. These rankings consolidate ICT's standing as a strong method for both continuous and discrete tasks.

A.3 Hyperparameter Setting

We report the details of hyperparameters in our experiments. The number of iterations, T , is set to 200 for continuous tasks and 100 for discrete tasks. For most continuous and discrete tasks, we employ the Adam optimizer [46] to fine-tune the proxies. The learning rates are set at 1 e -3 and 1 e -1 for continuous tasks and discrete tasks, respectively. In the case of the Hopper Controller task, the input dimension is significantly larger, at 5126 , and we adopt a smaller learning rate 1 e -4 for fine-tuning to ensure stability of the optimization process. Regarding the learning rate for the meta-learning framework, we use the Adam optimizer [46] with a learning rate 2 e -1 for continuous tasks and 3 e -1 for discrete tasks, respectively.

A.4 Analysis of Co-teaching and Sample Reweighting Efficacy

In our analysis, we focus on two key steps of our method: (1) pseudo-label-driven co-teaching and (2) meta-learning-based sample reweighting. We evaluate the efficacy of these steps by comparing generated samples with their corresponding ground truth. It's important to note that during the training phase, ground-truth scores are inaccessible to all algorithms and are used strictly for evaluation. Our method incorporates three proxies f θ 1 ( · ) , f θ 2 ( · ) , and f θ 3 ( · ) . We employ f θ 1 ( · ) for pseudo-labeling and f θ 2 ( · ) , f θ 3 ( · ) for co-teaching. We run ICT over 50 time steps for both D'Kitty (continuous) and TF 8 (discrete) tasks.

Pseudo-label-driven co-teaching. The step involves selecting 64 samples with smaller losses for fine-tuning the proxies while ignoring the remaining 64 samples. To assess the effectiveness of this strategy, we calculate L Sel , the mean squared error (MSE) between the pseudo-labeled and ground truth scores of the selected 64 samples, and L Ign , the MSE for the ignored samples. These calculations are averaged over 50 steps. We find that for D'Kitty, L Sel is 0 . 124 lower than L Ign and for TF 8 , it's 0 . 006 less than L Ign . These results validate the efficacy of this step, as the selected samples more closely align with the ground truth.

Meta-learning-based sample reweighting. In this step, we aim to assign larger weights to cleaner samples and smaller weights to noisier ones among the total of 64 samples. We measure the efficacy of this step by calculating L Large , the MSE between the pseudo-labeled and ground-truth scores of the 32 samples with larger weights, and L Small , the MSE for the 32 samples with smaller weights. These calculations are averaged over 50 steps. We observe that for D'Kitty, L Large is 0 . 010 lower than L Ign . For TF8, L Large is 0 . 005 less than L Small . These findings indicate that the samples with larger weights are indeed closer to the ground truth, substantiating the effectiveness of this step.

A.5 Comparison with RGD

We conduct a deeper analysis comparing our sample reweighting method with the RGD method, as proposed in [26]. The RGD method, drawing inspiration from distributionally robust optimization, aims to provide an efficient and theoretically sound way to reweight samples during training.

To ensure a fair comparison, we perform additional experiments, keeping the dataset and experimental conditions consistent for both methods. Two benchmarks, TFB8 and D'Kitty, are evaluated.

Our results in Table 7 indicate that while RGD is efficient and provides a theoretically effective solution, our approach delivers slightly superior performance. This performance edge might stem from our method's capability to utilize information from the static dataset.

A.6 Examining Hyperparameter Sensitivity Further

a)

(b)

0.90

0.90

0.80

0.80

Score

β=4e-1

β=8e-2

K=32

K=16

0.70

β=le-1

β=2e-1

β=5e-2

0.60

0.60

0.50

0.50

0

25

50

75

100

0

25

50

75

100

Step T

Step T

(

(d)

0.90

0.95

0.80

α=4e-3

α=8e-4

Score

α=4e-3

α=8e-4

α=2e-3

α=5e-4

0.90

α=2e-3

α=5e-4

S

α=1e-3

α=1e-3

0.60

0.85

0.50

0

25

50

75

100

0.80

0

50

100

150

200

Step T

Step T

Building on the analysis from Sec 4.6, we delve deeper into hyperparameter sensitivity, focusing on the TF8 task. Specifically, we investigate the influence of the number of selected samples ( K ) in the first step, and the learning rate ( β ) in the second step.

Figure 5 (a) displays the 100 th percentile normalized ground-truth score as a function of the time step T for different K values ( 8 , 16 , 32 , 64 ). ICT demonstrates stability over a specific range for varying K values, showcasing its robustness. Notably, ICT reaches optimal designs around t = 20 and maintains this level, further validating its resilience against different optimization steps T .

Figure 5 (b) plots the 100 th percentile normalized ground-truth score as a function of the learning rate ( β ) in TF 8 . ICT maintains a consistent performance across diverse β values, corroborating its robustness concerning the hyperparameter β in TF 8 .

Furthermore, we evaluate the effect of the fine-tuning learning rate α in both TF8 and D'Kitty tasks. Figures 5 (c) and 5 (d) reveal a consistent performance across varied α values for both tasks, highlighting ICT's robustness towards the fine-tuning learning rate.

A.7 Limitation

We validate the effectiveness of ICT across a broad spectrum of tasks. Nevertheless, certain evaluation methodologies do not completely represent authentic situations. For instance, in the superconductor task [5], we adhere to the established convention of utilizing a random forest regression model as the oracle, in line with previous studies [1]. Regrettably, this model may not perfectly mirror the complexities of real-world cases, resulting in discrepancies between our oracle and the ground-truth. Future collaborations with domain experts can potentially refine these evaluation methods. Overall, given the straightforward formulation of ICT, combined with empirical proof of its robustness and effectiveness across diverse tasks in the design-bench [1], we maintain confidence in its capability to effectively generalize to other scenarios.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel model-based optimization method that combines ensemble-based ideas and meta learning. The authors introduce three proxy models, each capable of generating pseudo-labels for the training of the other models, effectively making them co-teachers. This co-teaching process serves as an ensemble training method, mitigating potential risks associated with pseudo-labeling. Additionally, the paper incorporates meta learning techniques by re-weighting the samples, assigning higher weights to accurate samples. The intuitive nature of this idea and the reasonable performance observed in the experiments make a compelling case for accepting this paper.\n\nSTRENGTHS:\n1. There are several methods available for stabilizing the learning process with pseudo-labels, and this idea has the potential to be applied to various literature in the field.\n\n2. The idea presented in this paper is straightforward and effectively communicated, allowing readers to grasp its essence easily.\n\n3. One of the key contributions of this paper is the thorough analysis of hyperparameters, which play a critical role in model-based optimization. The authors provide valuable insights into optimizing these parameters to achieve better performance.\n\nWEAKNESSES:\n1. Although this paper presents a highly intuitive method, it lacks a detailed mathematical explanation for why this approach is effective. It would be beneficial to include analysis based on experiments to provide more insight into the working mechanism.\n\n2. It appears that several ideas in this paper draw inspiration from the prior method of BDI [1], particularly in the context of offline model-based optimization. While this viewpoint is intriguing, a more explicit analysis comparing the proposed method with BDI would greatly assist readers in understanding the similarities and differences.\n\n3. It would be valuable to reference offline biological design methods [2,3], even though they may not directly address continuous tasks. Specifically, [2] can be good reference to discuss this method due to their utilization of pseudo-labeling and their success in stabilizing the bootstrapping process, which aligns with the idea presented in this paper. Including a dedicated discussion section on these references would greatly benefit future researchers.\n\n[1] Chen, Can, et al. \"Bidirectional learning for offline infinite-width model-based optimization.\" Advances in Neural Information Processing Systems 35 (2022): 29454-29467.\n\n[2] Kim, Minsu, et al. \"Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequences.\" arXiv preprint arXiv:2306.03111 (2023).\n\n[3] Jain, Moksh, et al. \"Biological sequence design with gflownets.\" International Conference on Machine Learning. PMLR, 2022.\n\nQUESTIONS:\n1. The paper employs three proxy co-teachers, but it would be interesting to investigate the impact of reducing the number to just two proxies. Are there any notable observations or differences in performance with this modification?\n\n2. The meta learning process appears to involve bi-level optimization of parameters and weights. Considering the potential complexity of this approach, it raises concerns about computational resources required to execute the code. It would be helpful to provide insights or strategies to address this issue.\n\n3. While this method demonstrates impressive performance in continuous design tasks, its effectiveness in discrete tasks, such as TFBind8, seems to be limited. How does this method perform on other protein tasks like GFP or UTR? Is there any intuition as to why this method excels specifically in continuous tasks?\n\n4. The method's performance, when measured by the median score (50th percentile), appears to be weaker in comparison to the maximum score (100th percentile). Can you provide some intuition or explanation for this observation? It would be valuable to understand the factors contributing to this difference in performance.\n\nLIMITATIONS:\nThis paper really already gives limitations which I really appreciate it.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new method to address the out-of-distribution issue of offline model-based optimization.\nThe author first generates a pseudo-labeled dataset and identifies valuable (important) data for proxy co-teaching.\nTo alleviate potential inaccuracies in pseudo-labels, a meta-learning framework is proposed to adjust the importance weights of samples.\nExperiments on both continuous and discrete tasks demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The out-of-distribution issue in offline model-based optimization is a challenging problem.\nThe proposed method is well-motivated, technically sound, and interesting. \n\n2. Experiments and ablation studies on both continuous and discrete tasks demonstrate the effectiveness of the proposed method.\n\n3. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. In Lines 150-151, the authors claim that \"small-loss samples typically contain valuable knowledge, making them ideal for enhancing proxy robustness\". However, in my opinion, these small-loss samples might be some samples easy to learn since different proxies generate similar outputs for them. These easy samples may not be the truly valuable samples that benefit model generalization.\n\n2. In Section 3.2, a meta-learning framework is proposed to adjust the importance weights of samples. However, there still exist distributional shifts between the inner-loop and outer-loop. Therefore, the loss in Equation 7 may not be informative enough to identify truly important samples for generalization. Samples with hight weight might be the ones similar to the offline dataset. Therefore, the proposed method may not be applicable when the distributional shift is large.\n\nQUESTIONS:\nIs it possible to visualize the so-called valuable samples such that the intuitive interpretation can be obtained?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the offline MBO problem, which aims to synthesize an instance $x$ that maximizes the proxy $f(\\cdot)$ estimated by a neural network. The popular approach, gradient ascent, updates the instance $x_{t+1}$ to have a higher proxy value compared to the previous instance $x_t$. However, this approach encounters a challenge where the proxy $f(\\cdot)$ becomes less accurate as the iterations progress, as the instance $x_t$ becomes more out-of-distribution.\n\nTo overcome this challenge, the paper proposes two techniques inspired by the field of noisy labels. Firstly, the paper applies co-teaching, a method that filters confident samples by selecting those consistent across two models. In this case, since the labels are generated by a neural network, the final method involves three models: one pseudo-labeler and two co-teachers. The three models are trained alternately, changing their roles. Secondly, to further enhance performance, the paper introduces sample reweighing, a technique that assigns higher weights to confident samples using meta-learning. When combined, the proposed ICT (importance-aware co-teaching) method outperforms previous approaches.\n\nSTRENGTHS:\n- The idea of applying noisy label techniques to handle pseudo-labels of OOD samples makes sense.\n- The proposed two components are necessary, as shown in the ablation study.\n\nWEAKNESSES:\n### Experiments are not convincing\nMy biggest concern is the experimental results. The gains over the prior works are marginal compared to the high variances. The rank seems better at first glance, but it is not really convincing as the gap of proxy scores is too incremental.\n\n### On the evaluation metric\nI know this is not the problem of this paper but of the entire field. Still, reporting only the proxy value as the evaluation metric may not be enough. How can one confirm that the updated instance $x_T$ follows the valid form of $\\mathcal{X}$?\n\n### Limited technical novelty\nThe proposed method is basically an application of known methods such as co-teaching and sample reweighing. Although the three model parts were somewhat new, thanks to the use of the pseudo-labeler, the technical innovation is not sufficiently strong.\n\n### Presentation can be polished\nThe writing contains various repetitions, with similar words appearing in the abstract, introduction, method, and conclusion. It would be great if the paper could provide more information and prune redundant details. Additionally, Figure 1 and 2 are hard to understand without reading the full texts. The captions can be polished to be self-contained.\n\nQUESTIONS:\nWritten in the weakness.\n\nLIMITATIONS:\nDiscussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work proposes an Importance-aware Co-Teaching for Offline Model-based Optimization (ICT) as a solution for Offline model-based optimization problems. The ICT algorithm is comprised of 2 steps: (i) pseudo-label-driven co-teaching and (ii) per-sample reweighting.\n\nOverall the approach aims to learn an ensemble of methods where the targets predicted by one is treated as the pseudo-label for the other models. Furthermore, the more confident samples with low losses are used to give feedback to the proxy model.\n\nSTRENGTHS:\n* The paper proposes their intuition backed by two clearly laid out steps to walk the readers through the working of the model.\n* The extensive results on multiple datasets of both continuous and discrete domains help better bolster their approach.\n* The ablation of each step is useful to better weigh each design choice of ICT\n\nWEAKNESSES:\n* Some of the results in Table 1, Table 2 are within standard deviation. I would advice the authors to also perform a p-test to better strengthen their claims.\n* It would be nice to expand more on the per-sample weighting part. My knowledge on this subject often uses a validation data to as a proxy for minimizing loss, similar to Meta-weight-Net [1]. In the current training regime, it is not clear to me why the model cannot converge to the trivial solution of a zero vector as per-sample weights.\n\n[1] Shu, Jun, et al. \"Meta-weight-net: Learning an explicit mapping for sample weighting.\" Advances in neural information processing systems 32 (2019).\n\nQUESTIONS:\nI am interested in understand two design choices taken in this work:\n\n* In co-teaching, the two models other than proxy share knowledge about top-K points which both have learnt very well. If the K points are in-distribution and all models have learnt them perfectly, I see little to gain from this step. I was thinking using the worst-K points might be more interesting, as I am not sure how the training dynamics would play out there.\n* I per-sample weighting, i am curious how the current proposed regime fares compared to other solutions such as RGD [2] which have been shown to achieve improved generalization inspired by DRO literature. RGD would also remove the need to learn another meta-network  However, this again focusses on giving more importance to samples the model is weak at, and roughly connects back to the first point.\n\n[2] Kumar, Ramnath, et al. \"Stochastic Re-weighted Gradient Descent via Distributionally Robust Optimization.\" arXiv preprint arXiv:2306.09222 (2023).\n\nLIMITATIONS:\nThe approach is in the domain of optimization, and does not directly have any negative impact by itself. The authors have provided a nice paragraph covering this aspect in their paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}