RoMA: Robust Model Adaptation for Offline Model-based Optimization
Authors: Sihyun Yu
Sungsoo Ahn
Le Song
Jinwoo Shin
Venue: NeurIPS 2021 Poster (2021)
Keywords: model-based optimization

Abstract: We consider the problem of searching an input maximizing a black-box objective function given a static dataset of input-output queries. A popular approach to solving this problem is maintaining a proxy model, e.g., a deep neural network (DNN), that approximates the true objective function. Here, the main challenge is how to avoid adversarially optimized inputs during the search, i.e., the inputs where the DNN highly overestimates the true objective function. To handle the issue, we propose a new framework, coined robust model adaptation (RoMA), based on gradient-based optimization of inputs over the DNN. Specifically, it consists of two steps: (a) a pre-training strategy to robustly train the proxy model and (b) a novel adaptation procedure of the proxy model to have robust estimates for a specific set of candidate solutions. At a high level, our scheme utilizes the local smoothness prior to overcome the brittleness of the DNN. Experiments under various tasks show the effectiveness of RoMA compared with previous methods, obtaining state-of-the-art results, e.g., RoMA outperforms all at 4 out of 6 tasks and achieves runner-up results at the remaining tasks.


RoMA: Robust Model Adaptation for Offline Model-based Optimization
Sihyun Yu!
Sungsoo Ahn
Le Song2.3
Jinwoo Shin
KKorea Advanced Institute of Science and Technology (KAIST)
2 Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) BBioMap
{sihyun. (kawoss)@kaist..oo [Peter.AM Ahn, Le. ac.
ac
Abstract
We consider the problem of searching input maximizing black-box objective function given a static dataset of input-output queries. A popular approach to solving this problem is maintaining proxy model, e.g., deep neural network (DNN), that approximates the true objective function. Here, the main challenge is how avoid adversarially optimized inputs during the search, Le., the inputs where the DNN highly overestimates the true objective function. To handle the issue, we propose a new framework, coined robust model adaptation (RoMA), based on gradient-basee optimization of inputs over the DNN. Specifically, it consists of two steps: (a) a pre-training strategy to robustly train the proxy model and (b) a novel adaptation procedure of the proxy model to have robust estimates for a specific set of candidate solutions. At a high level, our scheme utilizes the local smoorhness prior to overcome the brittleness of the DNN. Experiments under various tasks show the effectiveness of RoMA compared with previous methods, obtaining ofte--of-hhe-at results, e.g., RoMA outperforms all at 4 out of 6 tasks and achieves runner-up results at the remaining tasks.
Introduction
Designing new objects with the desired property fundamental problem in wide range of real-world domains, including chemistry [13], biology [5, 41], robotics [3, 28]. and aircraft design [20]. Unfortunately, many of them often require evaluating an expensive objective function, e.g., synthesizing and measuring the fluorescence of the protein [5]. Model-based optimization (MBO) is powerful framework circumvent this issue, based on an approximation of the expensive evaluation of objective functions by cheap proxy model. Specifically, MBO optimizes solution with respect to the proxy model instead of the costly objective function. For obtaining a better proxy model, predominant works have typically assumed active (or online) learning scenarios, i.e., they allow for updating the model interactively with new queries of choice to approximate the ground-truth objective function better around optima [22, 37, 39, 40, 43, 45, 46).
Recently, researchers have also proposed offline versions of MBO algorithms [5, 8, 11, 25, 52] which build the proxy model from a given static dataset, i.e., input-output pairs collected from the objective function prior to building the proxy model. Such frameworks are beneficial for various real-world scenarios where the new interactive access to the objective functions may accompany serious danger or expensive cost for evaluation, e.g., drug discovery or aircraft design. The common challenge be addressed for offline MBO is to construct proxy model that yields accurate approximation of the
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

Solution update
D
Gaussian noise
Optimization
Model adaptation
Offline dataset
(E
fax 0)
f(a;0))
Weight perturbation
Step 1. Pre-training
Step adaptation solution update
Figure 1: Overall illustration of our proposed robust model adaptation (RoMA) framework.
Wrong solution
Accurate solution
Without local smoothness prior
With local smoothness prior
Figure 2: Illustration of the effect of employing local smoothness prior.
true objective function outside the training dataset, as the optimization procedure mostly leads the solution to be out of the dataset and causes severe overestimation due to domain shift [12, 48].
Intriguingly, Fu and Levine [11], Trabucco et [52, 53] found the gradient-based optimization of solutions to be effective for offline MBO. Such procedure is attractive since it can utilize gradient information the proxy model, e.g., a deep neural network (DNN), generate fine- ine-grained solutions. However, due to the highly non-smooth nature of DNNs, such gradient-based optimization may generate adversarial examples [51] whose score deviates significantly between the DNN-based proxy model and the true objective function. Existing works have attempted to regularize this behavior using conservative training [52] or normalized maximum likelihood [11]. In this paper. we investigate more direct way regularize such non-smooth nature of DNN.
Contribution. We introduce new framework, coined robust model adaptation (RoMA), for offline MBO to optimize the solution via gradient- based updates from the DNN proxy model. Our main idea utilize local smoothness prior for alleviating the brittleness of DNN at adversarial examples. To be specific, RoMA consists of two steps: (a) a pre- treiiraining strategy to robustly train the proxy model and (b) a novel adaptive adjustment procedure of the proxy model to have robust estimates for specific set of candidate solutions. We provide overall illustration of our framework in Figure 1.
(a) We first introduce a pre-training procedure of the proxy model to locally smooth at inputs and eventually becomes robust to the gradient-based input change. The novel component of the pre- training procedure is an additional regularization to force a flat loss landscape with respect to the model parameters, rather than only employing regularization based input- level perturbation into the training objective. Such consideration the flatness to the model parameters is effective in improving the robustness of the proxy model the gradient-based updates.
(b) To strengthen the accurate update of solution candidates from the trained proxy model, we also present a novel adaptive adjustment strategy of the proxy model to enhance the local smoothness inputs outside the training data. Specifically, candidates by adding a small perturbation the model weights After that, an update of solution candidates occurs from the adjusted model rather than the original DNN proxy model. We discover that the proposed adjustment strategy aptly mitigates the fragility of the proxy model at pradient-basedd updates to given inputs and improves the performance. We also emphasize that the adaptive strategy is worthwhile in offline MBO as the expressive power of the proxy model is limited under the fixed number of data and parameters; it is difficult sufficiently regularize all inputs with the domain shift from the dataset without any adjustments.
propose adjust the model given solution

---

Algorithm Robust model adaptation (RoMA)
1: for to K do
P Step 1. Pre-training with weight perturbations. Find 0 <- arg mex0) (.x.)))D.55 [[[(xr 0; y)²] via PGD.
3: Reparameterize 0 using O such that 02 O2 de for some fixed Φr and & 1,..., L.
4: Minimize F( [f(( + 6; 0) y)² over 0 using gradient-based update.
5: end for
6: Set 0 +- 0 and choose (0) from the training dataset DD
7: for to T
Step Optimizing the solution with iterative model adaptation. [= 0)1|2 z-a(()) + a(f(x ;0) fxx('; ((=();0--1) via PGD.
8: Find 0: +- arg ((
9: Compute r. (1+1) from x using gradient-based update.
10: end for
Output the final solution (T))
We extensively verify the effectiveness of our RoMA under Design-Bent BennhBBechhhP [53], an offline MBO benchmark diverse domains such as chemistry, biology, and robotics. In particular, achieves the best result in 4 out of 6 tasks when compared to strong baselines [5, 8, 11, 25, 52]. For all tasks, RoMA achieves the normalized 100th percentile score' of 1.705, while the second-hest baseline [11] achieves 1.687. Intriguingly, our framework dramatically improves the 50th percentile scores, demonstrating how it generates diverse solutions with superior quality.
method
the remaining tasks, RoMA at least achieves the runner-up results. When averaging
2
RoMA: Robust model adaptation
2.1 Overview of RoMA
To describe our framework, we define our problem follows. Consider real-valued vector R2,, e.g., protein sequence, and a real-valued objective function f*(x) E R, e.g., fluorescence of a protein sequence or the morphology of the robot. We are interested in finding optimal input xx E R² that maximizes the objective function, ie.,, x= = arg max (x)). Rather than having access to the objective function, we are given a static dataset D = {(,,, ) of input-output pairs (I,, y1) queried from the objective function, i.e., 1/1
(a:)-
Since the true objective function is unknown, our RoMA framework approximates using surrogate model fa: 9) parameterized as a deep neural network (DNN) with weights 0. Then one can use gradient-based updates to find fine-grained input maximizing the surrogate model. However, without any regularization, DNNs are prone overfitting and naïve gradient ascent algorithms can easily find adversarial examples whose score deviates significantly between the surrogate model and the true objective function. Our main contribution lies in resolving this issue by regularizing the smoothness of the surrogate model, i..., decreasing the sensitivity of the model with respect to input perturbations.
Such a local smoothness prior for regularizing DNNs has been successfully used to enhance the generalization of DNNs in various scenarios, e.g., adversarial robustness, generative models, and uncertainty estimation [1, 7, 9, 14, 29, 31, 35, 38, 47, 51, 56, 57]. Especially, we note the striking resemblance between our framework the adversarial training methods [7, 51, 56]; they seek to prevent the existence of adversarial examples generated from tradient-based optimization of inputs. In this respect, one may expect the smoothness prior to beneficial for our offline MBO framework.
To be specific, our RoMA is a two-stage framework, described as follows.
Step 1. Train the proxy model on the dataset D to approximate the true objective function with Gaussian smoothing of inputs under worst-case weight perturbations.
Step 2. For time-steps 0,..., T proxy model to be locally smooth at the solution (1))
;
update the solution (x()) after adapting the output of the
We provide the detailed description of our framework in Algorithm 1.
'The percentile score is computed from 128 solutions proposed by the offline MBO algorithms.

---

In the rest of this section, we explain each component of RoMA in detail. In Section 2.2, we describe how the RoMA pre-trains the surrogate model with Gaussian smoothing (Step 1.). In Section 2.3, we explain the procedure of updating the solution while simultaneously adapting the proxy model to yield input-level smoothness at the intermediate solution (Step 2.). Finally, Section 2.4, we describe some other algorithmic components in RoMA, such as an approach to handling discrete inputs or a procedure to select hyperparameters without the true objective function.
2.2 Pre-training with weight perturbations
We first describe our pre-training strategy for the DNN-based proxy model f(x; θ) to approximate the true objective function f (z) on training dataset D. We focus on regularizing the training with Gaussian smoothing of inputs under worst-case weight perturbations [55]. To this end, we optimize an T-llayered DNN with weight as the proxy model to minimize the approximation loss as follows:
(
max @E
((a.))D,6--0000 [[f(= 5; - -)
where 5 is random variable sampled from the Gaussian distribution (O, o²) with mean 0 and standard deviation ø. Furthermore, B(0) is the set of perturbed parameters of the DNN that are near the parameter 0, defined as follows:
B(0) == 0 10 - Oell S 1.-110ele vee (1........1)
Here, 8 >> indicates the maximum magnitude of the perturbation, Og is the parameter the f-th layer and ||-Ilæ denotes the Frobenious norm of matrix. To solve the inner maximization problem in (1), we utilize the projected gradient descent (PGD) [32], similar to Wu et al. [55]. We remark that our pre-training uses Gaussian noise for input perturbation, while Wu et al. [55] uses a bi-level worst-case input perturbation. We found that using the Gaussian noise leads to a much more stable training for our problem. We provide more details of this optimization procedure in the Appendix A.
Such an input-level Gaussian smoothing, i.e., augmenting inputs with additive noise sampled from Gaussian distribution, has shown to effectively regularize the smoothness of DNNs in various domains [27, 30] and moreover improve the adversarial robustness [7]. Furthermore, training the model under the worst-case perturbation of weights induces a flat loss landscape with respect to weights, which has shown to be beneficial for generalization of the training objective outside the
training dataset [6, 10, 21, 26, 36]. In particular, Stutz et al. [49] and Wu et al. [55] empirically observed a strong correlation between generalization capability of the model robustness at adversarial examples and the flat loss landscape of the model parameters.
2.3 Iterative proxy model adaptation
Once the proxy model is pre-trained to approximate the true objective function, one may apply gradient-based updates the intermediate solution 2i(t) for time-steps
0, O.... T 1, e.g.. ((t+1) (f) V 0)l ())))())- However, during the pre-training stage, apply Gaussian smoothing to the proxy model only for the training dataset D; the gradient-based updates may be erroneous for intermediate solutions far away from the training dataset D, i.e., inputs where the model is not smooth. To circumvent this issue, at each time step I, we adjust the output of the proxy model to be smooth at the intermediate solution (,()) before update these candidates. Such a model-adjustment procedure effectively focuses the model's limited expressive power for the input ((()) This strategy can also be interpreted as a test-time adaptation procedure [54] which adapts to information of the input (1) before estimating the true objective function f (x)()).
Formally, at time-step f, we search for the model parameter 0 at the current solution candidate (1) with the following objective, where the objective is optimized via the PGD method similar to the inner-maximization problem of (1):
0, = arg min 0E
|/V./(x; 11 z=n(t)
+ aa - 1(x0) -
where we let 0-1 = 0 and > 0 is the hyperparameter control the output of the proxy model changing too much from the regularization. We also constrain the model parameter O, to be in the

---

local neighborhood (()) of the original pre-trained parameter. This adjustment allows the proxy model to maintain good approximation for the true objective function since the corresponding regression loss is lower-bounded for 0 E B()) from the training objective in (1). Note that we smooth the proxy model based minimizing the input gradient norm [1, 9, 14, 47] rather than Gaussian smoothing as in Section 2.2; we found such a regularization to be more effective as directly regularizes the gradient used for updating the solution. Here, one may ask why only the current intermediate solution (²()) is considered for the adjustment, rather than incorporating other inputs, e.g., the previous solution (=(-1), 1), This is because we are interested in local behavior at the ((()) for updates, so only considering z0(0) to force the local smoothness is enough for the adjustment.
2.4 Other algorithmic components
Discrete inputs. Since our framework uses gradient-based based updates on continuous- inputs, it is non-trivial to extend for discrete-valued inputs. To resolve this issuc, we propose to use a variational autoencoder (VAE) [24] which maps discrede-valued input to a continuous-
latent vector 2. To be specific, we train an encoder network g(x) and a decoder network M(2)) using the VAE objective. Next, we train a proxy model [=; 0) which operates
the latent space to approximate the true objective function, i.e., f o g(æ) ~ (x). Then we can optimize over the space of latent vectors using RoMA with respect to the proxy model f(z; 0). After optimizing the latent vector for T steps to obtain 2 2(2), we sample from the decoder h(z(()) to report the discrete-valued solution.
Hyperparameter selection strategy. Offline MBO algorithms aim to generate highly scoring solutions under the constraint of not accessing the true objective function. Unfortunately, there is no rule of thumb for selecting hyperparameters in such an offline setting [53], and it is an active area of research. Nevertheless, we provide guidelines for selecting the hyperparameters of our RoMA in the offline setting. For the maximum magnitude E of weight perturbations, we choose the largest value such that the pre-training of the proxy model in Section 2.2 is stable. For choosing T and a, we set it to a constant value which is task-agnostic. Indeed, we empirically observe the performance of RoMA to be robust under different choice of such constant values.
3 Related work
Offline model-based optimization. Offline model-based optimization (MBO) has recently gained interest for objective functions that are expensive or dangerous to evaluate. To be specific, existing offline MBO methods generate candidate solutions based on either sampling from a generative model or gradient-basee updates to a sample existing in the training dataset. For the former approach, Brookes et al. [5], Fannjiang and Listgarten [8] train variational auto-encoders [24] along with the proxy model to designate a trust-region where the optimized solution via generation under such region is accurate. Moreover, Kumar and Levine [25] learns inverse mapping from the scores to the inputs via conditional generative adversarial network [34]. However, these approaches require a specific tuning across different domains to learn valid manifold of high-scoring inputs.
Instead, several works have focused on the latter approach of utilizing gradient- rrasient-based updates to optimize the solution. Specifically, Fu and Levine [11] maximizes the normalized maximum likelihood estimate of the true objective function, and Trabucco et al. [52] regularizes the proxy model to assign lower scores to the sample-generating policy. However, in these gradient-based methods, the non-smooth nature of deep neural networks has been overlooked. We directly tackle this issue by regularizing the smoothness of the model via proposing two-stage framework.
Regularizing smoothness of deep neural networks. Smoothness regularization network (DNN) has been regarded as a trustworthy regularization to enhance its generalization capability in wide range of domains [38, 57], e.g., image classification [31, 47], adversarial robustness [7, 32, 56, 58), generative models [1, 9, 14, 35], and uncertainty estimation [29]. They have employed various forms of regularization that to mitigate the highly non-sme non-smooth nature of DNNs. For instance, Loshchilov and Hutter [31], Guirajani et al. [14] suggest to directly regularize the weights of the DNN and Sokolić et al. [47] proposes to minimize the spectral norm of the DNN parameter. Moreover, Arbel et al. [1], Fedus et al. [9]. Sokolić et al. [47] give an input gradient penalty while training. Another line of works, especially under the context of the adversarial robustness in image classifier, they usually attempt to regularize the smoothness by considering an input-level noise,
deep neural
5

---

like Gaussian [7] or worst-case noise [32. 58] that maximizes the training objective. Inspired by these success, our method employs the smoothness prior into the offline MBO for better generalization of the model with its robustness to adversarial inputs.
Test-time adaptation. Test-time adaptation has shown considerable improvement to handle the distributional shift from the training data distribution at test-time [2. 18, 50, 54]. These approaches propose an objective to adapt the model at test-time to perform better at the specific task under unseen data, e.g., robust image classification against corruptions [19] or generalization of reinforcement learning policy across different environments [18]. Similarly, we design a model adaptation objective so that the update of the given new solution candidate becomes robust.
4 Experiments
We verify the effectiveness of our framework on Design-bench [53],² an offline model-based opti- mization (MBO) benchmark consisting of 6 tasks with various domains. Our result exhibit that the proposed robust model adaptation (RoMA) framework significantly improves the overall performance in diverse offline MBO tasks compared to recent offline MBO frameworks [5, 8, 11, 25, 52]. We also perform ablation studies to validate each component in our RoMA.
We follow the same setup for all experiments in prior works for the evaluation [11, 52, 53]: we optimize and propose 128 solution candidates as the final solution and evaluate their 100th/50th percentile scores. We report the numbers obtained by the previous works unless otherwise specified. All the numbers are averaged over 16 runs along with the standard deviations for reporting our results.
Baselines. We consider 6 prior approaches in offline MBO as baselines for comparison: CbAS [5], Autofocus [8] NEMO [11], MINS [25], COMs [52], and naive gradient ascent (Grad. Ascent) [53].
Implementation details. We use a 3-layer multi-layer perceptron (MLP) in all experiments, with the width size 64 and the softplus activation function. Adam optimizer [23] with the learning rate of 0.001 utilized to pre-train the proxy model with the dataset of each task. Gradients are clipped by norm of 1.0, and we set the mini-batch size 128. For variational auto-encoder (VAE) [24), use the same architecture as the one used in prior works to utilize VAE for offline MBO [5. 8). For the optimization of solutions, we select inputs that have the highest score from the dataset as initial candidates, like prior works that utilize gradient-based methods to optimize solutions [11, 52]. For optimizer update solution candidates, processed with 4 GPUs (NVIDIA RTX 2080 Ti) and 24 instances from a virtual CPU (Intel Xcon Silver 4214 CPU @ 2.20GHz). and it takes at most 44 hours to run each task over 16 runs.
also utilize Adam optimizer [23]. All the experiments
Hyperparameters. For the main experiments, we set the number of solution update to be large enough, i.e., T = 300. For the coefficient for regularization a, we & across all tasks. We choose the largest maximum magnitude of a weight perturbation E so that the pre-training of the 0.0005 for GFP, Molecule, and Superconductor
proxy model is possible. Specifically, we set task and choose 5 = 0.005 at other 3 tasks. For the step size 7), we decay proportional to the output of the model at the current solution from the initial value; See the Appendix B for more details.
Tasks. In what follows, we describe the tasks from offline MBO benchmark Besign-Benche [53].
The objective of green fluorescence protein (GFP) is to find protein that has high fluores- cence, which is proposed by Sarkisyan et al. [41]. The dataset consists of a total 5000 number of proteins with fluorescence values, and each protein is composed of 238 amino acids, i.e., 238 discrete variables where each variable indicates 20 categorical one-hot vectors.
the Molecule task, one finds substructure of a molecule that has high activity at the test against the target assay [13]. The dataset involves 4,216 data points, where each data consists of 1,024 discrete variables, and each variable indicates the Morgan radius 2 substructure fingerprints.
The Superconductor (Supercond.) task aims to find a superconducting material with a high critical temperature, where the dataset is provided by Hamidieh [17] and consists of 21,263 data in total. Each data an 81-dimensional vector with the corresponding critical temperature.
In the HopperController (Hopper) task, one optimizes the parameter of the policy neural network to maximize the expected return on the Hopper-v2 task in OpenAl Gym [4]. Each
http:a/wwwa....eaaanoaoaaaoeeeee bench

---

Table I: Comparison of 100th percentile scores for each task. We mark the scores within standard deviation from the highest average score to be bold.
Discrete domain
Continuous domain
Method
GFP
Molecule
Supercond.
Hopper
Ant
Dkitty
Avg.
Dataset Max
3.152
6.558
73.90
1361.6
108.5
215.9
1.000
CbAS [5]
0.408
6.301=0.131
72.17
547, 11423.9
393.0+3.750
396.1:60.65
1.324
Autofocus [8]
3.365=0.023
6.345=0.141
77.07=11.11
443,8
386.9:10.58
376.3:447.77
1.286
NEMO [11]
3.359=0.036
6.682=0.200
127.0=7.192
2130.11556.
196.11
431.61147.79
1.687
MINs [25]
3.315=0.029
6.508
80.23=10.67
746. 13636.8
18..54
352.9138.65
1.304
COMs [52]
3.305=0.024
6.87620.128
110.0=66.0f
2395.71.61.7
378.8
341.4:28.47
1.589
Grad. Ascent [53]
2.894=0.001
6.63620.066
89. 69.20000
(050.8a2245.
399.911190 9111.911111
390.71:49.24
1.237
RoMA (Ours)
3.007
6.890
103.9a5.487
2466,5
168.5
384.3151.68
1.705
Table 2: Comparison of 50th percentile scores for each task. We mark the scores within one standard deviation from the highest average score to be bold.
Discrete domain
Continuous domain
Method
GFP
Molecule
Supercond.
Hopper
Ant
Dkitty
Avg.
Dataset Max
3.152
6,558
73.90
1361.6
108.5
215.9
1.000
CbAS [5]
3.269::00188
1.472=00113
32.21
232.5+23.88
267.3=16.55
203.2:3.580
0.826
Autofocus [8]
3.216:20029
5.759=0.158
31.57=7.457
116.4:
176.7=59.41
199.3
0.752
NEMO [11]
3.219:0.039
5.814::0.022
66.41 4.618
390.2:44.37
326.9=5.229
18..8
0,960
MINs [25]
3.135
5.806=0.07s
37.32 10.50
520.4+:30.5
184.8=29.55
211.6:13.67 13.67
0.803
Grad. Ascent [53]
2,
2.894:00000
6.401=0.186
54,06=5.06
185.0:72.88
318.0=12:05
6.3.3
0.862
RoMA (Ours)
3.23010.015
6. 1.100101
68.99
560.1:22.47
370.8=6.771
252.4
1.103
datapoint is pair of policy neural network weights of 5,126 parameters and its expected return simulated by a simulator from Brockman et al. [4], and the size of the dataset is 3,200.
The goal AntMorphology (Ant) and Dkitty Morphology (Dkitty) tasks to find the optimal robot parameters, e.g., positions and orientation of robot joints. Each input is 56 and 60- dimensional vector and the dataset size 12,300 and 9,546, respectively.
Following prior offline MBO methods [11, 52] that have relied on Design- for evaluating their algorithms, we remark that experiments are done on Design-bench, which consists of the set of tasks using MLP. Nevertheless, we emphasize that Design-bench still includes diverse scenarios (discrete and high-dimennsionl inputs) with important applications (biology, chemistry, material design, and robotics), which is enough to solve general MBO problems.
4.1 Main results
In Table and 2, we report the 100th and 50th quantile scores of the solutions found by our RoMA and the baselines, respectively. We also report the average of normalized score across 6 tasks, i.e., average of 2 Vesn where y, Moaxs Busn indicates the score of proposed solution from our RoMA framework, maximum and minimum values in the dataset, respectively.
Wota
As reported in Table 1, RoMA outperforms the AntMorphorlogy task with large margin and shows a state-of-the-ar result at Molecule, HopperController, and DkittyMorphology tasks. Moreover, we note that RoMA achieves at least runner-up results at all 6 tasks, i.e., framework shows the competitive results at all tasks to other prior approaches. Finally, our method shows the state-of-the-an result compared to prior works
average of normalized scores.
Intriguingly, RoMA also shows consistent improvements and shows competitive results on both of discrete tasks, namely Molecule and GFP. Here, we note that most prior methods suffer to achieves high 100th percentile scores both tasks. confirms how the VAE-based mapping from discrete inputs to latents is effective for leveraging gradient-basel offline MBO methods to discrete inputs.
7

---

7.0
ClAS
120
CLAB
MINS RoMA (Cher)
Astudacuu
Asoet
Autafocus
NEXO
RoMA (Dam)
Critical temperatore
NEMO
100
Actiity
6.6
80
0.4
6.2
6.0
100th
19th
98th
97th
96th
96th
100th
80th
Percentile
(a) Molecule
(b) Superconductor
Figure 3: Scores of the ground truth objective function evaluated at samples different percentiles in Molecule and Superconductor task. The dotted lines indicate the maximum score in the dataset.
We also note that RoMA dramatically improves 50th percentile scores compared to existing ap- proaches, as shown in Table 2; our framework outperforms all at 50th percentile scores at 4 out of 6 tasks and is at the second place for the remaining tasks. Moreover, the 50th percentile scores of of 6 tasks are larger than the maximum value in the dataset, which supports the superiority of our proposed method reliably improving the solutions using gradient-based updates.
4.2 Ablation studies
Finally, we perform ablation studies be specific, we start by analyzing the ratio of found solutions that outperforms the best output in the dataset. We then verify the effect of each component of RoMA, namely (a) the pre-training strategy (b) and the model adjustment procedure. In the end, we conduct experiments to show the stability of the framework a different choice of hyperparameters.
framework to validate the effect of each component. To
Ratio of high-scoring solutions. One can apply the naive baseline of selecting the highest- input in the training dataset for our optimization problem the solution. Accordingly, it is worthwhile to analyze the ratio solutions from each algorithm that outperform such a naïve baseline. To this end, in Figure 3, we provide scores corresponding various percentiles for the Superconductor task and the Molecule task. Since prior works have only reported 100th/50th percentile scores for each task, we re-implemented and report other percentile scores for comparison.
In this respect, we extensively figure out the ra- tio of those outperforming solutions among entire candidates moreover to the 50th percentile score. Results in Figure 3 demonstrates RoMA better op- timizes solution candidates to other previous ap-
2750
Expected retazzi
2500
2250
proaches with the higher ratio. Precisely, the 70th
percentile score in Superconductor task and the 96th percentile score in Molecule task from our framework is still larger than the maximum score from the dataset, while existing approaches fails to optimize the solution at such a percentile.
2000
RoMA
RoMA w/o adaptation.
1500
Gaussian smoorhing
0
100
200
300
400
500
Effect of each component. We start with ablating the effectiveness of each component in RoMA. To this end, we compare RoMA with (a) RoMA with- out the model adaptation scheme and (b) RoMA without both the model adaptation and the robust pre-trate trainriiing schemes. To be specific, (a) pre-trains the proxy model using the pre- training strategy but optimizes the solutions without the model adaptation. Next, (b) trains the proxy model only using the input-level Gaussian smoothing.
Step
Figure 4: Illustration of ablation study
HopperController task to analyze the behavior of each component in our framework.
Figure 4 summarizes the result of the comparison for the Hopper task. Remarkably, replacing the naive pre-training into our proposed strategy pre-training significantly boosts the performance. Employing the model adjustment gives additional improvement. This confirms how the pre-

---

7.1
7.0
Ohitball temperatur
3
6.9
200
3.30
6.8
(A)
80-
(3)
(3)
(8)
3.20
(C5
(D)
6.6
100
200
500
930
500
100
200
300
400
000
100
200
3000
003
500
Shep
Step
Step
(a) GFP
(h) Molecule
(c) Superconductor
500
400
Dipected exture
2503
600-
Expueter meturn
3250
125
(A]
(4)
.000
1750
630
(3))
(80
375
200
1003
(D)
2000
2003
300
400
500
100
200
500
430
100
200
300
400
500
Step
Stopp
Step
(d) HopperController
(c) Antoorphology
(1) MorthMoophologg
Figure 5: Comparison of 100th percentile scores to different hyperparameter choices in all 6 tasks. All experiments are averaged over 16 runs. Here, (A), (B), (C), and (D) indicates the result from different choices of hyperparameters (a, 8): (ooo Eo), (ao, 0 0.2ep), (0.1ag,50) and (0.1og, 0.28p), respectively. Here, and En denote the hyperparameters used in our main experiments, e.g., Table 1.
training and the adjustment strategies bring orthogonal improvements in our framework. We also provide ablation study each term of the adaptation objective; See Appendix E for details.
Hyperparameter selection. We remark the challenge in offline MBO to select the hyperparameter, as we do not have any access to the objective function. Accordingly, analyzing the effect of hyper- parameter choices in the offline situation crucial and worthwhile. this respect, we empirically verify the effectiveness of our framework regarding hyperparameter choices. Specifically, we show our framework stable a different choice of the number of iteration T and the coefficient a the adjustment. Moreover, we also validate the strategy for selecting the maximum weight perturbation magnitude e. To be specific, we provide curves for each task with T 500, to validate the stability of our framework to the more number of iterations. For regularization coefficient a at the model adjustment, we compare the result a = 0.1 and in all tasks. In the end, we compare the result from the determined from our procedure and the smaller magnitude 0.2s.
Figure 5 illustrates the result. We first note that the performance on all tasks stable even if we choose larger T for optimizing the solution. In the case of a, the result is quite stable at the different choice of a, which verifies choosing universal a is enough our framework. Moreover, the result is better or at least similar at choosing the largest compared to the one with smaller confirming our hyperparameter selection strategy on finding from the offline static dataset. Here, we emphasize that we do not select hyperparameters with the best performance; we use systematic strategy for hyperparameter selection that only relies on observations made during training. Consequently, the performance can be even better with different hyperparameters several tasks; See Appendix E.
5 Discussion and conclusion
We propose new framework for offline model- optimization (MBO) using deep neural networks (DNNs) called robust model adaptation (RoMA). Specifically, we derive two-step procedure based on employing local smoothness prior to alleviate the fragility of the DNNs outside the training data, as the optimization procedures cause domain shift from the given dataset. We emphasize that this approach is the straightforward regularization to handle such brittleness of the model. Extensive experimental results demonstrate the superiority of algorithms across a variety of tasks, both the best solution and the overall quality of whole solution candidates.

---

Limitations and future works. Although RoMA shows its effectiveness on design problems in the offline MBO benchmark, which contains a wide range of domains [53], we note that the evaluation of optimized solutions is not accomplished by actual experiments. Instead, the assumed ground-truth function utilized for the evaluation is synthetic, e.g., robot simulator or pre-trained DNNs with a large number of the dataset. As the world of nature is often more complicated, exploiting our framework in real-world situations may face unexpected challenges and not be suitable, unlike synthetic tasks. However, we believe RoMA can still be beneficial even in these cases, similar to the success in reinforcement learning to build a framework in the simulator- apply such algorithm to actual robots [16]. Still, verifying and employing the framework in more realistic, practical situations with domain experts is a worthwhile and intriguing future research direction, and we leave it for future work. Finally, we remark that the evaluation of RoMA is done using multi-layer perceptron; applying our approach into more complicated architecture and data is also interesting future direction, e.g., image optimization with convolutional network architectures.
environment and successfully
Negative social impacts. The automatic design of objects can serve crucial role in mitigating diverse safety and economic issues in real situations, e.g., aircraft design or drug discovery. Although most prior works have focused on its positive aspects, we note that such a role may be either malicious: if the desired property is the one that hurts individuals. For instance, the proposed framework can be utilized for automatically optimizing biochemicals that threaten people; it shares a similar input space search molecules proteins for drug discovery, but only the purpose is the opposite side. In this respect, one should be aware of the drawback of improving offline MBO algorithms.
Acknowledgments and Disclosure of Funding
SY thanks anonymous reviewers, Jaeho Lee, Sangwoo Mo. Soojung Yang, Seokhyun Moon, and Jaeyoung Park for their helpful feedback on the early version of the manuscript. This work was partially supported by Institute of Information & Communications Technology Planning & Evaluation (ITP)) grant funded by the Korea government (MSIT) (6.22019-000007, Artificial Intelligence Graduate School Program (KAIST)). This work was mainly supported by Samsung Research Funding & Incubation Center of Samsung Electronics under Project Number SRFCIT1902-06.
References
[1] M. Arbel, J. Sutherland, M. a. Binkkowski, and A. Gretton. On gradient regularizers for mmd gans. In Advances in Neural Information Processing Systems, 2018
[2] P. Banerjee, т. Gokhale, and C. Baral. Self-supervised test-time learning for reading com- prehension. In North American Chapter of the Association for Computational Linguistics, 2021.
[3] F. Berkenkamp, A. Krause, and A. P. Schoellig. Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics. arXiv preprint arXiv: (602.04450, 2016
[4] G. Brockman, V. Cheung, L. Pettersson, Schneider, J. Schulman, Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv: 1606.015400 2016.
[5] D. Brookes, H. Park, and J. Listgarten. Conditioning by adaptive sampling for robust design. In International Conference on Machine Learning, 2019.
[6] S. Cicek and S. Soatto. Input and weight space smoothing for semi-supervised learning. International Conference on Computer Vision Workshop. 2019.
[7] J. Cohen, E. Rosenfeld, and Z. Kolter. Certified adversarial robustness via randomized smooth- ing. In International Conference on Machine Learning, 2019.
[8] C. Fannjiang and Listgarten. Autofocused oracles for model-based design. In Advances in Neural Information Processing Systems, 2020.
[9] W. Fedus, M. Rosca, B. Lakshminarayanan, A. M. Dai, S. Mohamed, and I. Goodfellow. Many paths to equilibrium: Gans do not need to decrease a divergence at every step. International Conference Learning Representations, 2018
10

---

[10] P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur. Sharpness aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
[11] Fu and S. Levine. Offline model- based optimization via normalized maximum likelihood estimation. In International Conference on Learning Representations, 2021.
[12] Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. In Interna- tional conference on machine learning, 2015.
[13] A. Gaulton, L. J. Bellis, A. P. Bento, J. Chambers, M. Davies, A. Hersey, Y. Light,
S. McGlinchey, D. Michalovich, B. Al-Lazikani, et al. Chembl: large-scale bioactivity database for drug discovery. Nucleic acids research, (IDD)::110--DIII
2012.
[14] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems, 2017.
[15] T. Haarmoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, 2018.
[16] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint 2018.
[17] K. Hamidich. A data-driven statistical model for predicting the critical temperature of superconductor. Computational Materials Science, 154:346-354, 2018.
[18] N. Hansen, R. Jangir, Y. Sun, G. Alenyå, P. Abbeel, A. A. Efros, L. Pinto, and X. Wang. Self supervupe policy adaptation during deployment. In International Conference on Learning Representations, 2021.
[19] D. Hendryeks and T. Dietterich. Benchmarking neural network robustmess to common corrup- tions and perturbations. In International Conference on Learning Representations, 2019.
[20] W. Hoburg and P. Abbeel. Geometric programming for aircraft design optimization. AIAA Journal, 52(1 (2(11))24442226 2014.
[21] N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang. On lurge-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2016.
[22] C. KI Williams. Gaussian processes formachine learning. Taylor & Francis Group, 2006.
[23] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International 2015.
Conference on Learning Representations, 2015.
[24] D. P. Kingma and M. Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
[25] A. Kumar and S. Levine. Model inversion networks for model-based optimization. In Advances in Neural Information Processing Systems, 2020.
[26] H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein. Visualizing the loss landscape of neural nets. In Advances Neural Information Processing Systems, 2018
[27] Y. Li, Xu, and F. Liu. Whiteout: Gaussian adaptive regularization noise in deep neural networks. stat, 1050:5, 2016.
[28] T. Liao, G. Wang, B. Yang, R. Lee, K. Pister, S. Levine, and R. Calandra. Data-efficient learning of morphology and controller for a microrobot, In International Conference on Robotics and Automation, 2019.
[29] J. 7. Liu, Z. Lin, S. Padhy, D. Tran, T. Bedrax- and B. Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. arXiv preprint 2001002006..1108 2020.
11

---

[30] R. G. Lopes, D. Yin, B. Poole, J. Gilmer, and E. D. Cubuk. Improving robustness without sacrificing accuracy with patch gaussian augmentation. arXiv preprint arXiv:1906.0m 2202::9900.0211,,, 2019.
[31] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Reprexentations, 2019.
[32] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
[33] E. J. Martin, V. R. Polyakov, X.-W. Zhu, L. Tian, P. Mukherjee, and X. Liu, All-assay-max pqsar: Activity predictions as accurate as four-concentration ic50s for 8558 novartis assays. Journal of chemical information and modeling, 59(10)4455 4459, 2019.
[34] M. Mirza and S. Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411, 1784, 2014.
[35] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
[36] B. Neyshabur, S. Bhojanapalli, D. Mcallester, and N. Srebro. Exploring generalization in deep learning. In Advances in Neural Information Processing Systems, 2017.
[37] J. Peters and S. Schaal. Reinforcement learning by seward-weightt regression for operational space control. In Proceedings of the 24th international conference Machine learning, pages 745-750, 2007.
[38] M. Rosca, T. Weber, A. Gretton, and S. Mohamed. A case for new neural network smoothness constraints. arXiv preprint arXiv:2012.07969 2020.
[39] R. Y. Rubinstein. Optimization of computer simulation models with rare events. Journal of Operational Research, 99(1):89-112, 1997.
European
[40] R. Y. Rubinstein and D. P. Kroese. The cross-entropy method: unified approach to combinato- rial optimization, Monte-Carlo simulation and machine learning. Springer Science & Business Media, 2013.
[41] K. S. Sarkisyan, D. A. Bolotin, M. V. Meer, D. R. Usmanova, A. S. Mishin, G. V. Sharonov, D. N. Ivankov, N. G. Bozhanova, M. S. Baranov, O. Soylemez, et al. Local fitness landscape of the green fluorescent protein. Nature, 5337766333397-441, 2016
[42] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization 2017.
algorithms. arXiv preprint 1
[43] B. Shahriari, K. Swersky, Z. Wang. R. P. Adams, and N. De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1), 2015.
[44] W.-J Shen. H.-. Wong, Q.-W. Xiao, X. Guo, and S. Smale. Introduction to the peptide binding problem of computational immunology: new results. Foundations of Computational Mathematics, 14(5):951-984 2014.
[45] J. Snock, H. Larochelle, and R. P. Adams. Practical bayesian optimization of machine learning algorithms, Advances in Neural Information Processing Systems, 2012.
[46] J. Snoek, O. Rippel, K. Swersky, R. Kiros, N. Satish, N. Sundaram, M. Patwary, M. Prabhat, and R. Adams. Scalable bayesian optimization using deep neural networks. In International conference on Machine Learning, 2015.
[47] J. Sokolié, R. Giryes. G. Sapiro, and M. R. Rodrigues. Robust large margin deep neural networks. IEEE Transactions on Signal Processing. 2017.
[48] C. Song, K. He, L. Wang, and J. E. Hopcroft. Improving the generalization of adversarial training with domain adaptation. In International Conference on Learning Representations, 2019.
12

---

[49] D. Stutz, M. Hein, and B. Schiele. Relating adversarially robust generalization to flat minima. arXiv preprint arXiv:2104.04442 2021.
[50] Y. Sun, X. Wang, Z. Liu, J. Miller, A. Efros, and M. Hardt. Test-time training with self- supervision for generalization under distribution shifts. In Proceedings of the 37th International Conference on Machine Learning. 2020.
[51] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intrigu- ing properties of neural networks. In International Conference on Learning Representations, 2013.
[52] B. Trabucco, A. Kumar, Y. Geng, and S. Levine. Conservative objective models: A simple approach effective model-based optimization. Advances in Neural Information Processing Systems Workshop, 2020.
[53] B. Trabucco, A. Kumar, X. Geng, and S. Levine. Design-bench: Benchmarks for data-driven offline model-based optimization, 2021.
[54] D. Wang, E. Shelhamer, Liu, B. Olshausen, and T. Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference Learning Representations, 2021.
[55] D. Wu, S.-T. Xia, and Y. Wang. Adversarial weight perturbation helps robust generalization. In Advances in Neural Information Processing Systems, 2020.
[56] C. Xie, M. Tan, B. Gong, A. Yuille, and Q V. Le. Smooth adversarial training. arXiv preprint arXiv:2006. 14536, 2020.
[57] Y. Yoshida and T. Miyato. Spectral norm regularization for improving the generalizability of deep learning. arXiv preprint arXiv: /705. 10941, 2017.
[58] H. Zhang, Y. Yu, J. Jiao, E. Xing, L. E. Ghaoui, and M. Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, 2019.
13


Reviews:
[{"review_id": "sZTHLs3flrP", "reviewer": "Reviewer_3o7x", "summary": "This paper proposes a gradient-based method that searches for inputs that maximizes the output of a (black-box) score function (model-based optimization).   In the analyzed scenario, the score function is an adversarially-robust pre-trained neural network, and the searching algorithm is gradient ascent (wrt to inputs) with a local regularizer in terms of the model parameters.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros.\n\n- The proposed method achieves good empirical results in continuous tasks in the Design-Bench benchmark.\n- The authors proposed solutions for both continuous and discrete inputs.\n\n\nCons.\n\nMy major concern is about the technical novelty and experimental setting/scope of the work.\n\n- Wrt to novelty, the pre-training strategy follows from existing work in the context of adversarial robustness. The iterative proxy model adaptation is \"gradient ascent\" with a regularizer that updates the model parameters based on a hand-designed objective. I appreciate (and agree up to some extent with)  the intuitive motivation provided for this objective. However, many other regularizers can be proposed based on a similar intuition. Since this is the major technical contribution of the work, and the experimental section considers only a single benchmark with a 3 layers MLP, I believe a better theoretical motivation/justification could be provided.\n\n- Experimental Setting/Scope: Many tasks can be formulated in the scope introduced in section 2.1 (line 74-79) some very challenging  (e.g. class-conditional image generation).  However, it is not clear why the tasks in the chosen benchmark should be considered over others. Would the proposed framework (RoMA) still be effective in more complex tasks and networks? If only such a benchmark is considered then the scope of the paper should be narrowed and motivation for the more narrowed scope should be provided.\n- Writing. I consider several parts of the writing that can be improved e.g. introduction and abstract could be improved.\n\nQuestions:\n\n- How important is regularizing the gradient norm vs keeping the output of the model close between steps? (e.g. first vs the second term in line 138) (This is not captured in figure 4).\n- Why model \"adaptation\" instead of model adjustment? In other words, is \"adaptation\" the right word?  If searching starts from  x_0 ~ D, and every step is only changing x  very locally why  x_1=x_0 + delta is necessarily another distribution. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "FxxMY51wW6I", "reviewer": "Reviewer_yVCx", "summary": "This paper proposes robust model adaptation, which first do a pretraining with adversarial weight perturbation. Then the final result is obtained by iteratively doing model adaptation on the learned model, with a local smoothness penalty objective. The proposed results shows improvements over existing methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is generally easy to follow, the motivation for the added smoothness penalty is clear. The pretraining step is very similar to what is proposed in [46], it would better if the authors could clearly illustrate the differences, if any. One concern is regarding hyper-parameter selection, as the authors have mentioned and also demonstrated in section 4.2. Although the method is relatively stable on several different hyper-parameter settings, the variance on some of the datasets seems large (i.e. Superconductor, HopperController). It seems that the selected alpha and eps values are the best performing ones. It may be good to include the results of other hyper-parameter settings in the main experiments so that it would be a more clear comparison, since the performance of the proposed method and other methods are close in most of the scenarios.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have described the limitations and impact in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "BboF4MdKXU", "reviewer": "Reviewer_XJN3", "summary": "RoMA is a framework that leverages the local smoothness prior to robustly approximate black-box objective functions. RoMA consists of two steps: 1, train the proxy models (neural networks) to approximate black-box objective functions by perturbing original inputs with Gaussian inputs and minimizing worst-case weight perturbations; 2, iteratively adapting the proxy model to enforce the local smoothness of loss landscape around the input x(t). This paper devises two separate loss functions for optimizing neural networks in these two steps. As a result, this method serves to avoid adversarial optimized inputs.\nIn the experiment section, this method is compared to the state-of-the-art methods on the Design-bench dataset. Results show that this method achieves state-of-the-art performance across a variety of tasks.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThis method involves two main steps, i.e., pre-training with weight perturbations and iterative proxy model adaptation. The first step leverages techniques from adversarial training; the second step utilizes the test-time adaptation technique. The combination of these two achieves good performance. In the related works section, literature in offline model-based optimization and regularizing smoothness of deep neural networks are summarized. There are two concerns. First, it would be helpful to give literature about test-time adaptation techniques related to the second step in this method; Second, it would be better to discuss the connections between previous methods and this method explicitly. For instance, what are the shortcomings of previous methods, and how does this method solve those shortcomings?\n\nQuality:\nOverall, this paper is technically sound:\n1.\tThe optimization objective functions devised for each step are sensible. It is clear how those objective functions could serve to achieve the desired property.\n2.\tThis paper’s experiments are comprehensive. The comparison between this method and other baselines on the Design-bench benchmark shows that this method could outperform previous methods across various tasks.\n3.\tThe ablation studies also demonstrate the necessity of each component in their method.\nOne concern is that since this method does not outperform other methods in all tasks, some analysis on why other methods in some tasks beat this method could better explain the potential limitations of this method. This paper is a complete piece of work. It presents the full idea with comprehensive experimental results to support its claims. This paper explicitly talks about the strengths, limitations, and future works in the last section. As for the weakness, considering that other methods on two Design-bench tasks outperform this method, some discussion on possible reasons for this would be appreciated. Also, besides the lacking of actual experiments limitation, more analysis on the weakness of this method is expected.\n\nClarity:\nThis paper is clearly written. The ideas behind each component are easy to understand. One concern is the term “input-level regularization”. Some brief definitions of this term would be appreciated. For the implementation details, this paper provides comprehensive implementation details in the experiment section. Also, details about each task and how they are evaluated are provided in both the paper and its supplementary materials.\n\nSignificance:\nThis paper achieves good performance in the black-box objective function estimation task. Their method is easy to understand. After comparing to other methods, RoMA achieves state-of-the-art results at some tasks and overall quality. As a result, other researchers and practitioners are likely to build their work on this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper includes comprehensive discussions on limitations, future works, and potential negative social impacts in the last section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "11"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper makes a somewhat incremental, yet solid and effective contribution to the area of model-based optimization by robust pre-training and a test-time robustification process. I concur with the reviewers and recommend the paper to be accepted at NeurIPS.
Decision: Accept (Poster)