Data-Driven Offline Decision-Making via Invariant Representation Learning
Authors: Han Qi
Yi Su
Aviral Kumar
Sergey Levine
Venue: NeurIPS 2022 Accept (2022)
Keywords: model-based optimization
offline learning

Abstract: The goal in offline data-driven decision-making is synthesize decisions that optimize a black-box utility function, using a previously-collected static dataset, with no active interaction. These problems appear in many forms: offline reinforcement learning (RL), where we must produce actions that optimize the long-term reward, bandits from logged data, where the goal is to determine the correct arm, and offline model-based optimization (MBO) problems, where we must find the optimal design provided access to only a static dataset. A key challenge in all these settings is distributional shift: when we optimize with respect to the input into a model trained from offline data, it is easy to produce an out-of-distribution (OOD) input that appears erroneously good. In contrast to prior approaches that utilize pessimism or conservatism to tackle this problem, in this paper, we formulate offline data-driven decision-making as domain adaptation, where the goal is to make accurate predictions for the value of optimized decisions (“target domain”), when training only on the dataset (“source domain”). This perspective leads to invariant objective models (IOM), our approach for addressing distributional shift by enforcing invariance between the learned representations of the training dataset and optimized decisions. In IOM, if the optimized decisions are too different from the training dataset, the representation will be forced to lose much of the information that distinguishes good designs from bad ones, making all choices seem mediocre. Critically, when the optimizer is aware of this representational tradeoff, it should choose not to stray too far from the training distribution, leading to a natural trade-off between distributional shift and learning performance.

Data-Driven Offline Decision-Making via Invariant Representation Learning
Han Qi", Yi Su", Aviral Kumar', Sergey Levine
Department of Electrical Engineering and Computer Sciences, UC Berkeley (han2019, aviralk)@berkeley. edu, yisuntvdgoogle. com ("Equal Contribution)
Abstract
The goal in offline data-driven decision-making is synthesize decisions that opti- mize a black-box utility function, using previously-collected static dataset, with active interaction. These problems appear in many forms: offline reinforcement learning (RL), where we must produce actions that optimize the long-term reward, bandits from logged data, where the goal is determine the correct arm, and offline model-based optimization (MBO) problems, where we must find the optimal design provided access only a static dataset. A key challenge in all these settings is distributional shift: when we optimize with respect to the input into a model trained from offline data, it is easy produce an out-of of-distribution (OOD) input that appears erroncously good. In contrast to prior approaches that utilize pessimism or conservatism to tackle this problem, in this paper, we formulate offline data-driven decision-make as domain adaptation, where the goal is to make accurate predic- tions for the value of optimized decisions ("target domain")) when training only on the dataset ("source domain"), This perspective leads invariant objective models (IOM), our approach for addressing distributional shift by enforcing invariance between the learned representations of the training dataset and optimized decisions. IOM, if the optimized decisions are too different from the training dataset, the representation will be forced to lose much of the information that distinguishes good designs from bad ones, making all choices seem mediocre. Critically, when the optimizer is aware of this representational tradcoff. it should choose not stray too far from the training distribution, leading to natural trade-off between distributional shift and learning performance.
1 Introduction
Many real-world applications of machine learning involve learning how to make better decisions from data. When we must make sequence of decisions (e.g., control a robot), this can be formulated as offline reinforcement learning (RL) [27, 30], whereas in problems where we must synthesize only one decision (e.g., a molecule that can effectively catalyze reaction), this can be formulated as multi- armed bandit or a data-driven model- -aaeed optimization (also known as black-box optimization [2]). In all of these cases, an algorithm is provided with data from previous experiments consisting of (x, y) tuples, where represents a decision (e.g., ... bandit arm, design such as molecule or protein, or a sequence of actions RL) and y represents its utility (e.g., certain property of the molecule or the long-term reward of trajectory). The goal is to generate the best possible decision, typically one that is better than the best one in the dataset. All of these problem domains are united by a common challenge: in order to improve, the decisions learned by the algorithm must differ from the distribution the designs in the training data, inducing distributional shift [23, 24], which must be carefully handled.
Current methods for solving such data-driven mecksign-making problems try learn some form of a proxy model f(:) that maps a decision x to its objective value y via supervised learning on the
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

static dataset, and then optimize the decision x against this learned model (in reinforcement learning. the proxy model is typically the action-value function and is learned via Bellman backups, thanks to the Markovian structure). However, overestimation errors in the learned model will erroneously drive the optimizer towards low-value designs that "fool" the proxy into producing a high values [23]. To handle this, prior works have proposed to regularize the learned model to be conservative by penalizing its values on unseen designs [25, 44, 49, 17], or explicitly constraining the optimizer (e.g..
by using density estimation [6, 23), referred to as "policy constraints" in offline RL [30]..
In this paper, we take different perspective for the distributional shift challenges in offline data- driven Drecision-making and formulate it as domain adaptation, where the "target" domain consists of the distribution of decisions that the optimization procedure finds, and the source domain corresponds to the training distribution. Framing the problem in this way. we derive class of methods that regulate distributional shift at the representation level, and admit effective hyperparameter tuning strategies. The intuition behind our approach is that, if the internal representations of the learned model are made invariant to the differences between the decisions seen the training data vs. the optimized distribution, then excessive distributional shift is naturally discouraged because excessive invariance would lead the out-of-distribution points to attain mediocre values on average. That is, as the optimized decisions deviate more from the training data, the requirement to maintain invariance will cause the model's 's representation to lose information, and will hence force the predicted value to become closer to the average y value in the dataset. This will force the optimizer to stay within the distribution of the dataset as recall that out-of distrribinttonn points will only appear mediocre under the learned model. Of course, the representation cannot be perfectly invariant in general, because the optimized points are different from the training data, and hence the method must strike a balance. In practice, one can instantiate this idea using an adversarial training procedure.
Fraction takre everage value of the dutsect outeide the traiing distrbutoon
Our main contribution is a new class of methods for offline data-driven decision-making making, derived from a novel connection with domain adaptation. In this paper, we instantiate this idea in the setting of of- fline data-driven model-based optimization (MBO) (i.e., offline data-driven bandits), leaving the sequen- tial offline reinforcement learning setting to future work. As diseussed above, our approach, invariant objective models (IOM), optimizes the decision in this case, "design" against learned model that admits invariant representations between optimized designs and the dataset. This enables controlling dis- tributional shift and also provides a way to derive effective offline hyperparameter tuning strategies that we show actually work well in practice. IOM can be implemented simply by combining any supervised regression procedure with a regularizer that enforces invariance. The regularizer is implemented using any discrepancy measure between two distributions, and we instantiate IOM using the x" discercpyyyy measure via the least-squares generative adversarial network. This procedure trains a discriminator to discrim- inate between the representations on the training dataset and optimized designs, whereas the learned representations are trained to be as invariant as possible to fool the discrininator. We theoretically derive IOM from our domain adaptation formulation of MBO and we also use this formulation to derive tuning strategies for several common hyperparameters. Empirically, we evaluate IOM on several tasks from design-Benncl [43] and find that it outperforms the best prior methods, and additionally, admits appealing offline tuning strategies unlike the prior methods.
Optiaal design within support
Ground-rath
Function value
Invastance
Conservatism
Function
valuas outiidie liph values on tmining dats
Duta
Figure 1: Illustration showing the intuition
behind IOM on a ID task. While conservatism (COMs) [44] pushes down the learned values on designs (decisions) not observed in the dataset and pushes up the learned values on designs (decisions) in the dataset, training with invariance induces the out-of of-distiibbttion designs (decisions) to attain ob- jective values close to the average value in the train- ing dataset. This discourages the optimizer from finding out-of-distribution designs (decisions).
2 Preliminaries
Offline data-driven decision-making and its variants. The goal in offline data- driven decision- making is to produce decisions that maximize some objective function, using experience from a provided static dataset. Two instances of this problem are offline reinforcement learning (RL., and offline data-driven model-based optimization (MBO). As shown Table 1, in offline RL, the decision is policy ı that prescribes action for any state s, and the objective function is the

---

Quantity
Offline RL
Offline MBO
Decision
policy I mapping states to actions
a design (e.g., molecule) the objective value. f(x
Objective function
long- long-term discounted reward, J(T)
=
Bag ~a [ 7'r(s, ax)]
Training dataset
dataset D {(TT, r)))= sampled a dataset D = ((x,, Ve }i= sampled from dist. over rollouts T, (U(T)
from the distribution of designs, (1(x) using find arg max Rxx (x))
Optimization problem
using DD find arg max; J(T)
Proxy model
learn model of (TT). typically in the form of a value function, Q(s, a)
learn model of f. f(x)
A typical method
optimize the learned Q-function Q
optimize the learned model 8
Distributional shift
shift between (aaas) and (Taas)
shift between μ((x) and (orr(x)
Table 1:
Instantiation of our generic formulation of offline decision-making in the context of offline RL
and offfine MBO. While the rest of the paper will adopt the terminology from oftline MBO, the ideas we present this paper can also be applied offline RL.
long-term discounted reward attained by the policy æ, denoted as ((T). In offline MBO, the decision corresponds to a design (x), and the objective function is the unknown black-box utility (f(x)). We will utilize the notation from offline MBO in this paper for simplicity, but the ideas developed in this paper can be extended to the RL setting.
Offline model-based optimization (MBO). As mentioned above, the goal in offline model-based optimization (MBO) [43] is to produce designs that maximize some objective function (x), by only utilizing a dataset D = {(x;, ) of designs x; and their corresponding objective values y:- No active queries to the groundtruth function are allowed. Typically offline MBÒ is formulated as finding the optimal design x". However, we will use a slightly more general formulation, where the goal is to optimize distribution over designs v(x). We will use this formulation largely for convenience of analysis, where the classic case is recovered when μ(x) is a Dirac delta function, though we also note that optimizing distributions over designs is often studied in settings such as synthetic biology, where the goal is to explicitly diversify the designs [5]. Thus, our goal is to find a distribution over designs porr (x) using only the static dataset D, such that it maximizes the expected value of (x) (the expected value under distribution y is referred to as J(a)). This can be formalized as follows (Alg refers to an algorithm))
pon == arg max Exxptt)) == (1) s.t. / ADg(D).
Aig
Model-based optimization methods. The methods we consider in this paper fit a learned model fo(x) to (x,, 31i pairs in the dataset via a standard supervised regression objective, with an additional regularizer. We will assume that fe is composed of two components: a representation e E KKd that maps given x into d-dimensional vector, and a forward model, fa that converts the representational output into scalar objective value, i.e., (x) == So (ox)). We will use the notation fo and fo interchangeably. Given learned model fo f)x())). one approach to obtain the optimized distribution porr by optimizing the expected value [p) := Ex= (f (ø(x)) under the learned model. This distribution (orr(x) can be instantiated in many ways: while some prior work [23, 6] represents нorт via a parametric density model over X' and optimizes this density model to obtain ('oe,, other prior work [44] uses a non non-parametric representation for μorh using a set of optimized "design particles"'
In this paper, we represent Morт using a set of design particles, each of which is obtained by optimizing the learned function fa (x(x)) with respect to the input x via grad. ascent, starting from i.i.d. samples from the dataset. Formally:
x + xx + V (Vxxo(6(x))) for & E [1, x) D
The optimized distribution pore is then given by porr = 6(x x²,
The challenge of distributional shift in offline MBO [23, 7, 9, 44]. When the learned model fo()(x)) is trained via supervised regression on the offline dataset D: arg ming 1
n E
2² the ERM principle states that [x()()) will reasonably approximate s(x) in expectation only under the distribution of the training dataset D, i.e., /Aaaaa however fo(ő(x;)) # ye in general on
points not the training distribution. Then, when optimizing the learned function, overestimation errors in the value of fa (0(x)) will inevitably lead the optimizer to converge to a distribution porr
3

---

over designs that have high value under the learned function(i.e., high Jo (HOrr)) but not a high ground truth value ((orr) ). In general, this distribution porr will be different from Mdatas since the learned function is likely to be more correct in expectation under Mdata-
3 Data-Driven Offline Model-based Optimization as Domain A Adaptation
In this section, we will formulate data-driven offline MBO domain adaptation. which will allow us to derive our method, invariant objective models (IOM). Instead of attempting to simply constrain the optimizer directly avoid distributional shift, our approach modifies the representations learned by the model fe by adding an invariance regularizer during training such that there is minimal distribution shift under the learned invariant representation during optimization.
If we can ensure that the learned model fe is accurate on the distribution found by the optimizer, Norr. then we can ensure that we will obtain good designs. Therefore, to handle the distributional shift, we can treat Adaaa as the "source domain" and the optimized distribution μort as the "target domain". Just as the goal in domain adaptation is to train on the source domain and make accurate predictions on the target domain, predictions under μort- Of course, learning to make accurate predictions on any unseen distribution impossible for any domain adaptation method [53], but in the case of MBO, the choice of the target distribution μоliт is also made by the algorithm itself, such that invariance can be enforced both by changing the representation and changing the target distribution Jorr.
will aim to train fo (x) on μ and with the goal of making more accurate
3.1 Invariant Objective Models (IOM)
IOM controls distributional shift between the optimized distribution Porr and the training distribution Maaa at the representational level, by regularizing the distribution of the features o(x) under forrt to match that under /Aauaa We will first present the intuition behind the method, and then formalize it by setting up a bi-level optimization problem that we will also analyze theoretically.
As the optimizer deviates from the training distribution jr, the invariance regularizer forces the distribution of optimized designs to resemble the distribution over representations ((x) under Pdata i.e., PP porr (d(x)) ss IP, pum (+(x)). When गorr is far from the training distribution (i.e., if DKL(PorT (daaaa is very high), it is easier to enforce invariance as it does not conflict with fitting fe on the training distribution. This representational invariance in turn causes the expected value of the learned function fo under Porr to appear (roughly) equal to the average objective value in the training dataset, i.e., (f(á(xx) = Ex-pan [fo (o())) if PoPt is too far away from the training distribution. As a result, the expected value of the learned model fo(((x)) under Horr, Jo (,oPT), will be worse than the value of the best design within the training distribution (since the best design input sampled from Mdat will attain a higher objective value than the average value under (.maa). This would act as guard to prevent the optimizer diverging too far away from the dataset, as out-of-distribution designs no longer appear promising. Iteratively regularizing & to enforce invariance between porr and ş until convergence will restrict the optimizer from selecting crroneously overestimated cott-o-ddisributtin inputs. We supplement this intuition with an illustration shown in Figure 9. We will now present formalization of this intuition in the form of a bi-level optimization problem that aims to maximize a lower-bound on the expected objective value attained under MoPt, and derive the training objectives for IOM using this bi-level optimization formulation. However, do note that, our practical method Algorithm IOM: Training and Optimization
Intuition:
does not aim to solve a bi-level optimization exactly due to computational infeasibility.
I: Input: training data D, number of gradient steps T == 50 to optimize por starting from the training distribution μ., training iteration K, batch size 2: Initialize: representation model (z(+). learned
Bi-level optimization for invariant objective models (IOM). To enforce invariance at a rep- resentational level, we utilize an invariance reg- ularizer. Denoted as disc (p, 4), our invariance regularizer is a discrepancy measure between two probability distributions p and q. w.r.t. certain hypothesis class H. For example, if H. is the class of all 1-Lipschitz functions, discH the 1-Wasserstein distance; if H is the class of linear functions, discæ is the maximum mean discrepancy (MMD) [13] distance. In practice, we implement this via generative adversarial network discuss in the next subsection.
model fo(.). set of optimized design partieles {x; mm
3:
for l k= K do,
4: Sample n training points (x,, ~ DD
5:
Run one gradient step (Equation 4) w.r.t Og and
du obtain Φe+1 and foc-, ($++1 (+).
6:
Run one gradient step to optimize design parti-
cles w.r.t. fo(o(x)): x + x V (xt)..
7:
end for
4

---

In addition to training fo via standard supervised regression on the labels in the training data, IOM trains the representation d(x) and the model f(-) to additionally minimize the discrepancy disc (x())), IP "oor
(o(x)) between distributions of representations under the optimized distribu-
tion, porr and the training distribution μs. This objective can be formalized as:
max Hame
Ex-por [fō (Ø" (x))
1 -((((x)) - yz)² + A disc pum (d(x)); non (o(x))) o.f
s.t. (z) arg min
where À denotes weighting hyperparameter. In Section 3.2, will describe how convert this bi-level optimization problem into a practical MBO algorithm that can be implemented with high capacity deep neural networks. This is followed by theoretical analysis to formally justify the objective in Section 3.3, and a discussion of how the hyperparameter can be tuned against a validation set in Section 4, without any online access the ground truth function.
3.2 Optimizing the Bi-Level Problem and the Practical IOM Algorithm
In this section, we will describe how we can convert the abstract bi-level optimization problem above into an objective that can be trained practically and discuss some practical implementation details. IOM alternates between solving the bi-level optimization problem with respect to the distribution μоrт and the learned model fe and @ independently. For training the learned model and the representation fo and o, IOM solves the inner optimization by running one-step of gradient descent (GD), utilizing the current snapshot of porr (denoted as (OPT):
Σ ((((((x))) - y1)² + x disc (M (o(x)), P Fom (o(x)))) .(4)
(dr+1, 0c+1) GDo
Simultaneously, we update the design particles representing /ÔM := 6(x) x2, towards optimiz- ing the learned function Soay2 (de+1 (x)): x + x + + V (1)+11 x Vi. Pseudocode for IOM shown in Algorithm
Practical implementation details: We model the representation o(x) and the learned function fo(+) each as two-hidden layer ReLU networks with sizes 2048 and 1024, respectively. We utilize the d'-discrepancy, which be instantiated in its variational form via a -quares generative adversarial network (LS-GAN) [32] the representations d(x). More implementation details be found in Appendix C.
3.3 Theoretical Analysis of Invariant Representation Learning for MBO
In this section, we will formally show that under certain standard assumptions, solving the bi-level optimization problem (Equation 3) enables to find a better design than the dataset distribution Musaa To show this, we will combine theoretical tools for analyzing offline RI. algorithms [8, 25, 50] and domain adaptation [53]. We show that the expected value of the ground truth objective under any given distribution π (x) of design inputs can be lower bounded in terms of the expected value under the learned function modulo some error terms. Then we will show that these error terms are controlled tightly when solving the bi-level problem, and finally appeal to uniform concentration to show that the performance of μоn is better with high probability. Our goal isn't to derive the tightest possible bound for IOM, but show that IOM attain reasonable performance guarantees, and this domain adaptation perspective can tackle distributional shift.
Notation and assumptions. Following standard assumptions [29]. we will analyze the setting when the learned representation lies in some space of functions, & E Φ and the learned function fo lies in fo E F. We will consider that the optimized distribution μоrт belongs to a function class, POPP E II. Our analysis will require that the ground truth model f is approximately realizable under some representation
& and for some y E F. That is, assume:
Assumption 3.1 (s reaaliizabillty) For any distribution æ over designs, there exists a representation and a g E F such that min, max Ex [If(x) - g(x(x))] - FF
In addition, we assume that Vʃ E Fl11/ < oo and is Lipschitz continuous with constant C: ||/ < C. This is true for most parameteric function classes, such as neural networks. Then, we can lower bound the average objective value under any distribution in terms of the invariance:
5

---

Propasition 3.2 ((nformal) bounding the ground truth value under π). Under Assumption 3.1 and the Lipschitz continuity on the function fo. the ground truth objective for any given II can be lower bounded in terms of the learned objective model So E F and representation & E 4, with high probability 2 1 thatt:
(T)) (TT)-J((m) 2 ((aan)) Jo((pana)) - - ddscc pan (((x)),PP(((()) -F - Estots
where Cy is a uniform constant depending on the function class F and Estat refers to statistical error that decays inversely with the dataset size, (D].
A proof for Proposition 3.2 is in Appendix A. The term can further be lower bounded using standard concentration error bound for ERM, and can be controlled if we train fo(o(-) minimize prediction error against ground truth values of f. Now we utilize Proposition 3.2 and a uniform concentration argument to lower-bound the value under the optimized distribution porr-
Proposition 3.3 ((Inormma)) Performance guarantee for IOM). Under Assumption 3.1, the expected value of the ground truth objective under por J (HOPT) is lower bounded by:
(1 log
| [
[
log
J(JORT) 2 H(Hdaa)) O
Jo(Joor) Jo Jo(palaa) (x)) (5)
(0)
A proof of Proposition 3.3 can be found in Appendix A. This proposition implies that long as the optimizer finds designs that improve the learned objective function fe E F. such that (o) is positive, and the discrepancy term (*) is minimized, then the distribution over designs found by IOM, μoFt, will compete favorably against the distributions of designs in the dataset. Of course, these bounds present the worst-case guarantees for IOM, but clearly improve over the best designs in the offlin dataset. In addition, this theoretical analysis also motivates the design of our scheme for performing hyperparameter tuning and model selection. This scheme will be based directly on the result from Proposition 3.2. We will discuss this connection when discussing tuning in Section 4.
will show in experiments, IOM does
Finally, we would like to highlight how invariance leads to "average" values if μorr deviates too far away from the data distribution μ.. If porr deviates too far, such that discH((* paa (ø), pon (0)) < <'. the learned function bounded too close to the average learned value on the data i.e., [[ [ I e'. Thus, the learned function simply reverts to "predicting the mean objective value" of the training distribution as e' decreases, thereby losing any information about the identity of the optimized distribution, when the optimizer goes too far out of μ.
4 Offline Workflow and Tuning for IOM
central challenge in offline model-based optimization that of offline workflow and hyperparameter tuning: not only must an algorithm be able to produce designs using a static dataset, but it must also prescribe a fully offine workflow for tuning hyperparameters so that the produced designs are as good as possible. This includes both explicit hyperparameters (such as the coefficient of conservatism or invariance, À for IOM) and implicit hyperparameters (such as the early stopping point). The primary challenge stems from the fact that unlike supervised learning, the final performance in MBO is determined by the optimized distribution μОPт, which is different from the training distribution, Adata- and we must reason about this performance fully offline.
In this section, we discuss how IOM admits a particularly convenient and fully offline hyperparameter tuning procedure that works well empirically. The goal of our tuning problem is to find a tuple consisting of the learned representation d², the learned model, fā. and the optimized distribution M out of given set of tuples obtained for different values of X. In other words, wish find the À from the set of models MM- (fa o², .u) ) such that the corresponding u2 attains the highest value under the ground truth objective. Crucially note that this must be done fully offline.
Our key idea: The primary question we wish to answer is: how can we identify if a given tuple (Sd: Φy, ((ee)) leads to good performance? In order for a tuple trained via TOM to perform well, we need two primary conditions: (1) it should be nearly invariant, and as result, robust to distributional

---

shift, and (2) the learned fe must attain good generalization performance within the training distri- bution. Intuitively (1) guards us against distributional shift, and (2) enables us find as good of a learned model on the dataset, that does not overfit or underfit as a result of the invariance regularizer. This forms the basis of our tuning method.
Offline tuning of loM: To instantiate the idea practically, we first run IOM with a set of hyperpa- rameters A (A1, .2...... Then for each run, we record the validation in-distribution error and the value of the invariance regularizer on a validation set. We then perform tuning in two steps: first, we filter all values that attain near-perrt near-perfect invariance under a user-specified margin E, i.e., disc uau (o^ (x)), 1P A.. (x))) S and are hence guaranteed to be robust to distributional shift. Second, we now pick models that attain good performance within the training distribution by selecting À values that attain the smallest validation prediction error: (fo (o(x)) - y)², in addition to picking the early stopping point based on the smallest validation error. This enables us to pick À that gives rise to a generalizing model fa d while being most robust to distributional shift. This still leaves a user-defined margin as free parameter, but we find that it comparatively insensitive and can be selected heuristically, as we show in our experiments in Section 5.2.
Formally, this procedure corresponds solving the following optimization:
X = arg min AEA
Σ (o y:)² s.t. discH (P (x)), P Puó (x)( Se 1=1
(6)
Finally, for more formal explanation of the soundness of this strategy. we note that applying Equation 6 provides a tight lower bound on the ground truth objective value in Proposition 3.2. Up to statistical error (which decays as [D] increases), the prediction error on a held-out validation set estimates
Proposition 3.2, and a smaller discrepancy (i.e., discæ S &) controls the (x) term in Prop. 3.2. Stat is statistical error, and EF & only depends on the function classes involved (F, t). This tuning À per Equation 6 can be justified as aiming to maximize a lower bound on ((on)).
The practical implementation is in in Appendix C.3.
In Section 5, we validate this offline tuming strategy, both against other alternative offline strategies for tuning IOM as well as offline strategies for tuning prior methods (COMs [44] and gradient descent [43]). We find that offline tuming strategy performs comparably to full online tuning without needing any online queries, whereas other strategies for IOM and other methods, lead to significantly poor performance compared to online tuning.
5
Experimental Evaluation
The goal of our empirical evaluation is to answer the following questions: (1) Does IOM outperform prior trate-of-the-- strategy for IOM better than other offline tuning strategies for IOM or other prior methods? (3) How sensitive is lOM to the choice of the hyperparameter X? In this section, we will answer these questions via comparative evaluation of IOM on several continuous data-driven design tasks from the standard tasks from the Design-Bench Bench suite [43].
methods for data-driven model-based optimization? (2) Is our offline tuning
5.1 Empirical Evaluation on Benchmark Tasks
We compare IOM to variety of prior methods for model-based optimization: CbAS [6], Autofo- cused CbAS [9] and MINs [23] that utilize generative models for constraining distributional shift; COMs [44] and RoMA [49] train robust models of the objective function by via conservatism and smoothness, respectively; gradient-free optimization methods such as REÍNFORCE [46), CMA- ES [15] and conventional Bayesian optimization BO-qEI [47]. We also compare the standard approach of learning a naïve model of the objective function, which is then optimized via gradient ascent. To better understand the benefits of enforcing invariance, we also compare IOM to IOM-C, which additionally applies a conservatism term on top of invariance that pushes up the learned values of designs in the dataset. Additionally, we add prior method that optimizes designs against lower-confidence bound estimate computed using a Gaussian process posterior in Appendix C.4.
Tasks. We evaluate four tasks with continuous-valuee input space from the Design- Bench [43] benchmark for offline model-bare based optimization (MBO). More details about these tasks can be found in Appendix C.
Evaluation protocol. Following prior work [43, 44, 49, 10], for each method, we query the learned model to obtain the top performing N points, i.c., X1,X,, ,Ny. We report the maximum objective
7

---

value among these N samples. This protocol reasonably reflects a real-world design process, where a number of computationally produced designs are tested and the best one is used for deployment.
Hyperparameter tuning. For fair comparison against prior methods studied in Trabucco et al. [43], we follow the uniform tuning strategy for reporting results in this section: we run lOM on each task for a given set of À values, and pick single value of À across every task. Note that this does require access to online evaluation, but prior works do use the same protocol. We will evaluate the efficacy of our offline tuning approach in Section 5.2 extensively.
Median
IQM
BO-qEI C CbAS MINs COMs
II
ASMAL CA Exby
IOM(ours)
[OM-C(ours)
0.87 0.90
0.93 0.96 0.87 0.90 0.93 0.96
Normalized Score
35%
100%
Choice of quandile
Figure 2: Left: Median and IQM [1] (with 95% Stratified Bootstrap Cls) for the aggregated normalized score on Design-bench. IOM improves over the prior methods, including those which use explicit conservatism although IOM does not. As we could not find the individual runs for RoMA [49], we report the mean and standard deviation for RoMA (copied from the results in Yu et al. [49]). along with more detailed results for other baselines in Appendix C.4, Right: The offline tuning performance matrix under different quantiles to filter the models based discriminator loss, compared with the oracle uniform tuning, which shows tuning strategy is robust to this hyper parameter.
Results. Following the convention proposed by Trabucco et al. [43], we evaluate all methods in terms of normalized score (higher is better). In Figure 2 (left). we show the median and interquantile mean (IQM) [1] for the aggregated scores across all tasks. Due to space limits, we only report the scores for the top- performing baselines (See Appendix C for the results for all baselines). IOM significantly outperforms all of the prior methods, including state-of-the-ar methods that employ conservatism, such as COMs [44]. This indicates that invariance alone (IOM) can effectively handle distribution shifts and avoid over-estimation for out-of -f-issiibittttion actions. Additionally, a comparison between the performance of IOM and IOM with conservatism (IOM-C) suggests that adding conservatism does not improve performance further. This further corroborates the fact that invariance is effective in preventing the optimizer from going out of distribution and conservatism is not needed. That said, we would like to clarify that this result does not mean that invariance will always be better than conservatism on any given offline MBO problem. In order to understand why IOM outperforms COMs on these tasks, we examine the in-distribution validation error of lOM and COMs in Appendix E and find that IOM learns less distorted functions that generalize better.
5.2 Analysis of the Offline Workflow for Tuning 1OM We will now present experiments to understand the effectiveness of our offline tuning scheme (Section 4) for tuning IOM.
It is important to highlight that, in general, offline tuning presents significant challenges for MBO. Prior works in bandits and reinforcement learning [52] have suggested that offline tuning should not work at all and, to the best of our knowledge, no prior work in offline MBO has proposed a principled and effective offline tuning method. In this section, we empirically validate our tuning strategy in comparison to other potential strategies, and compare results from our approach to oracle strategy that uses online evaluation of the true function.
Comparisons. We compare our offline tuning strategy to various offline strategies: (1). only tunes IOM based on validation-set prediction error, analogous to supervised learning (called "prediction error only"); (2). a strategy that tunes IOM based on only the value of the invariance regularizer (called "invariance only"): (3). a method that utilizes our tuning strategy for picking X, but not for picking the checkpoint ("uniform checkpoint selection") and two offline tuning strategies for tuning other prior methods: (a) tuning COMs [44] using validation-set prediction error (called "prediction error only"), conservatism loss (called "conservatism loss only") and both prediction error and conservatism loss, one by one (called "two-step tuning"); and (b) tuning an objective model trained via gradient ascent [43] using validation MSE error ("gradient ascent"). We report the drop in performance relative to the corresponding oracle online strategy that actually computes the "groundtruth" values of optimized designs in the simulator to choose hyperparameters in hindsight, i.c., the relative drop in
8

---

performance of the method (IOM, COM, grad. ascent) when tuned offline using the particular strategy compared to oracle tuning for the same method.
Results. We present our results in Figure (right) comparing the efficacy of various tuning schemes across various methods averaged over all tasks. While offinn tuning does lead to reduction in performance compared to oracle online tuning, we find that the drop is smallest for our offline tuning strategy applied to IOM. All other tuning strategies based on either only prediction error or the invariance regularizer lead to worse performance. This indicates that our tuning strategy is effective in tuning IOM. Furthermore, offline tuning strategies that utilize a validation set with prior methods, such as COMs and gradient ascent, also lead to a significant decrease in performance relative to the oracle. This is quite appealing as it suggests that IOM is more amenable to offline tuning strategies relative to prior offline MBO algorithms, Even if these prior methods can perform well when provided access to online evaluations, they might start to fail in real-world MBO problems when one needs to tune them fully offlin.. We discuss computational complexity of tuning the methods in Appendix E.
A Ablation analyses. Next we aim study the sensitivity of our offline tuning strategy with respect to the non-rigorous, user-specified parameter e in Equation 6 and checkpoint selection based on the validation prediction error. Figure (right) shows the offline tuning performance heatmap for all tasks (along their aggregated performance in the bottom row) under different choice of quantiles (where quantile 2% means the top x% of the models according to the value of the invariance regularizer, and the given value of were chosen). The color in each position of the matrix corresponds to the ratio of performance of our offline tuning strategy for given quantile value, compared to that of uniform oracle tuning. As shown in the bottom row, aggregated over all tasks, we can see our offline selection strategy is quite robust to the hyperparameter c. Figure (left) evaluates the efficacy of our checkpoint selection strategy, we plot the objective value J (HOPF) attained a function of the number of training epochs (i.e., checkpoints), alongside the values for the prediction MSE in the validation set for the corresponding epochs. The checkpoints selected by our early stopping scheme are shown via the vertical line. Observe that indeed the checkpoint for the lowest validation MSE corresponds to a rather good optimized distribution Morr for different tasks, no matter the optimal training epochs happen at the beginning or near the end, which validates our early stopping strategy. More detailed results showing the efficacy of early stopping for both IOM can be found in Appendix C.4.
Ant Morphology
Hopper Controller
Suboptimality Gap... 140 in
vattaron
200 Cenc antly
-225
Number of training MOGLTT
Number of training spocts
Figure 3: Left: Visualizing our checkpoint selection scheme two tasks, and matter the best checkpoint happens at the beginning or end of the training, our strategy based on validation in-distributtin MSE could capture the best performance. Right: Offline tuning vs oracle uniform online tuning for 1OM, COMs and Gradient-ascent. Among all the methods, IOM achieves the smallest regret for offline tuning, and our specific tuning strategy works best compared with other tuning strategies for IOM.
6 Discussion
In this work study offline data-driven model based optimization (MBO), where the goal is to produce optimized designs, purely using a static dataset. We proposed to view offline MBO through the lens of domain adaptation, and proposed a method that trains a model of the objective with an additional regularizer to promote invariance between the representations learned by the model in expectation under the training data distribution and the distribution of optimized designs. We show that this leads to a practically effective data-driven optimization method, and an appealing workflow for selecting the hyperparameters with offline data. Our method outperforms prior approaches, which illustrates the practical benefits of invariance. We also validate the effectiveness of the accompanying tuning strategy. While we evaluate our method on MBO problems, our approach can in principle also be extended to contextual bandits and even to, sequential reinforcement learning. But we admit that there might be some limitations: (1) fully data-driven offline optimization may on its own be insufficient (due to limitations of staying close to the dataset, hardness of tuning) and inevitably will require access to online evaluation. (2) a full understanding of theoretical conditions when invariance can outperform conservatism is important, but out of the scope of this work.
9

---

Acknowledgements
We thank Brandon Trabucco and Xinyang Geng for help in setting up the Design bench and COMs codebases. We thank anonymous NeurlPS reviewers, members of the RAIL lab at UC Berkeley and Amy Lu for informative discussions, suggestions and feedback on an early version of this paper. This research was supported by the Office of Naval Research, C3.ai, and Schmidt Futures, and compute resources from Google cloud. AK is supported by the Apple Scholars in AI/ML PhD fellowship.
References
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34, 2021.
[2] Christof Angermueller, David Belanger, Andreea Gane, Zelda Mariet, David Dohan, Kevin Murphy, Lucy Colwell, and Sculley. Population-based black-box optimization for biological sequence design. arXiv preprini arXiv:2006.03227, 2020.
[3] Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. In International Conference on Learning Representations, 2020. URL https: /ttp:////////eee/eeeeeeee net/forumidd 2irf
[4] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1): 151-175, 2010.
[5] Surojit Biswas, Grigory Khimulya, Ethan C Alley, Kevin M Esvelt, and George M Church. Low-n protein engincering with data-efficient deep learning. Nature methods, 18(4):389-39) 2021.
[6] David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. In Proceedings of the 36th International Conference on Machine Learning. PMLR, 2019. URL http::/proceediing/
mlr. pptps/////broooeeeeeeeee bbrooooe html.
[7] David H Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. arXiv preprint
2019.
[8] Ching-An Cheng, Tengyang Xie, Nan Jiang. and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. arXiv preprint vrXiv:2202.02446. 2022.
[9] Clara Fannjiang and Jennifer Listgarten. Autofocused oracles for model-based design. arXiv preprint arXiv:2006. 08052, 2020.
[10] Justin Fu and Sergey Levine. Offline model-based optimization via normalized maximum likelihood estimation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:
net/forum? FdFMMKK044....
[11] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1 1180-1189, PMLR, 2015.
[12] Ian Goodfellow, Jean Souget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. N1, 2014.
[13] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773 2012.
[14] Kam Hamidich. A data-driven statistical model for predicting the critical temperature of superconductor. Computational Materials Science, 154:346 354, 2018. ISSN 0927-0256. doi: http://dbi..rr//1.01016.. wwtmateei..0011mm....... htt.:///.........ooooooooooomoaa /ttaa////rrraceee/p/////////////
URL http: //www/ www.ssedieeencccccccoomm com/
[15] Nikolaus Hansen. The CMA evolution strategy: A comparing review. In José Antonio Lozano, Pedro Larrañaga, Iñaki Inza, and Endika Bengoetxea, editors, Towards a New Evolutionary Computation Advances in the Estimation of Distribution Algorithms, volume 192 of Studies in Fuzziness and Soft Computing, pages 75-102. Springer, 2006. doi: 10.100773-540-32444 LL. URL https: / /oi. org/ 10. 1007 / 32494-02244--111
10

---

[16]
Ashley Hill, Antonin Raffin, Maximilian Emesttss, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert. Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https: //gi //github.ar hamm/hilllasstaallle
//
basellleebaeeieee..... 2018.
[17] Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? arXiv preprint arXiv:2012.15083 2020.
[18) Fredrik Johansson, Uri Shalit, and David Sontag. Learning representations for counterfactual inference. In International conference on machine learning, pages 3020-3029. PMLR, 2016.
[19] Fredrik D Johansson, Nathan Kallus, Uri Shalit, and David Sontag. Learning weighted repre- sentations for generalization across designs. arXiv preprint arXiv: 08008, 2018.
[20] Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2013. URL. http:
/ //arxivv.av/ wwwgoogg//...o..o...o.....o. 0tbs/1312 1112..11.1111. 6114. cite arxiv: 1312.6114.
[21] Johannes Kirschner and Andreas Krause. Bias-robust bayesian optimization via ducling bandits. In International Conference on Machine Learning, pages 5595-5605. PMLR, 2021.
[22] Johannes Kirschner, lija Bogunovic, Stefanie Jegelka, and Andreas Krause. Distributionally ro- bust bayesian optimization. In International Conference on Artificial Intelligence and Statistics, pages 21844.. PMLR, 2020.
[23] Aviral Kumar and Sergey Levine. Model inversion networks for model-based optimization. NeurIPS, 2019.
[24] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pages 11761-11771, 2019.
[25] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04777 2020.
[26] Aviral Kumar, Amir Yazdanbakhsh, Milad Hashemi, Kevin Swersky, and Sergey Levine. Data-driven offline optimization for architecting hardware accelerators. arXiv preprint 10. 2021.
[27] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pages 45-73. Springer, 2012.
[28] Romain Laroche, Paul Trichelair, and Rémi Tachet des Combes. Safe policy improvement with baseline bootstrapping. arXiv preprint arXiv: 11177712222... 2017.
[29] Byung-Jun Lee, Jongmin Lee, and Kee-Eung Kim. Representation balancing offline model- based reinforcement learning. In International Conference on Learning Representations, 2021. URL
[30] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0 004444, 2020.
[31] Yao Liu, Omer Gottesman, Aniruddh Raghu, Matthieu Komorowski, Aldo Faisal, Finale Doshi- Velez, and Emma Brunskill. Representation balancing mdps for off- off-policy policy evaluation. arXiv preprint arXiv: 1805.000444 2018
[32] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2794-2802, 2017.
[33] Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Sungmin Bae. et al. Chip placement with deep reinforcement learning. arXiv preprint arXiv:2004. 10746. 2020.
[34] Willie Neiswanger and Aaditya Ramdas. Uncertainty quantification using martingales for misspecified gps. In Algorithmic Learning Theory, pages 963-982. PMLR, 2021.
[35] Quoe Phong Nguyen, Zhongxiang Dai, Bryan Kian Hsiang Low, and Patrick Jaillet. Value-at- risk optimization with gaussian processes. In International Conference Machine Learning, pages 8063-8072. PMLR, 2021.
[36] Thanh Nguyen-Tang. Sunil Gupta, A Tuan Nguyen, and Svetha Venkatesh. Offline neural contextual bandits: Pessimism, optimization and generalization. arXhv:2111.33807 2021.
11

---

[37] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable policy reinforcement learning. arXiv preprint arXiv: 1rXiv:1910000177, 2019.
[38] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.066 2017. URL http::/aaat aaxxvv..xvvvv/v/////////a///aaa org/ abs/1707 0ts//1777000647..
[39] Thiago D Simāo, Romain Laroche, and Rémi Tachet des Combes. Safe policy improvement with an estimated baseline policy. arXiv preprint arXiv:1909.0S5: 000999900..6,.... 2019.
[40] Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In Proceedings of the 25th International Conference Neural Information Processing Systems- Volume 2, NIPS 12, 2012. URL, http://dll. acm. org/ ci citation. cfm? .cm.org/
id=2999325 2999464.
[41] Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros. Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In Proceedings of the 32nd International Conference on Machine Learning. PMLR, 2015.
[42] Bharath K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Gert RG Lanckriet. On integral probability metrics,,phiiddveerss. hhiiics..ppiiiiiveeeeeee and binary classification. arXiv preprint arXiv:0901.2690 2009.
[43] Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-bench: Bench- marks for data-driven offline model-based optimization, 2021. URL https:////gthuutt com/ br aandonttaabuccodessig...
[44] Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Conservative objective models for effective offline model-based optimization. International Conference on Machine Learning, pages 10358-10368. PMLR, 2021.
[45] Martin J Wainwright. fiigh-dimensionll statistics: A non-asymptotic viewpoint, volume 48. Cambridge University Press, 2019.
[46] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist re- inforcement learning. Mach. Learn., 8:229-256, 1992. doi: 10.1007//BF00992266. URL https: setps://dois org/10. 10. 1007 /BF0999266.
[47] James T. Wilson, Riccardo Moriconi, Frank Hutter, and Marc Peter Deisenroth.
The reparameterizzation trick for acquisition functions. CoRR, abs/1712.00424, 2017. URL
http://arxivv http:///arxx..or//bbb/// 102.00000//11.00244....
[48] Liuyi Yao, Sheng Li, Yaliang Li, Mengdi Huai, Jing Gao, and Aidong Zhang. Representation learning for treatment effect estimation from observational data, Advances in Neural Information Processing Systems, 31, 2018.
[49] Sihyun Yu, Sungsoo Ahn, Le Song, and Jinwoo Shin. Roma: Robust model adaptation for offline model-based optimization. Advances in Neural Information Processing Systems, 34, 2021.
[50] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005.132399 2020.
[51] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. arXiv preprint arXiv:2102.08363. 2021.
[52] Andrea Zanette. Exponential lower bounds for batch reinforcement learning: Batch rl can be exponentially harder than online rl. In International Conference on Machine Learning, pages 12287-12297. PMLR, 2021.
[53] Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International Conference on Machine Learning, pages 7523-7532. PMLR, 2019.
12


Reviews:
[{"review_id": "nby6S2t8FnA", "reviewer": "Reviewer_Jwpc", "summary": " This work studies offline data-driven model based optimization (MBO), where the goal is to produce optimized designs, purely using a static dataset. We proposed to view offline data driven model-based optimization through the lens of domain adaptation, and proposed a method that trains a model of the objective with an additional regularizer to promote invariance between the representations learned by the model in expectation under the training data distribution and the distribution of optimized points. ", "questions": "Please refer to \"Weakness\".", "limitations": "They have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1.This paper studies the policy learning problem under the distributional shift. The total goal is to learn an invariant policy (representation) which maximizes some specific objectives (specific rewards) in the varying environments. Some related work is provided to support their novelty. \n\n2.Since the authors assumes the variability of marginal distributions (e.g., covariate shifts), they proposed the IOM method with MMD divergence between training and target features. Some theoretical results are provided to show the performance guarantee of the IOM method (a lower bound). \n\n3.Extensive experiments are provided to verify the effectiveness of the proposed method. Some ablation studies are provided with details.\n\nWeaknesses:\n\n1.I feel confused on the novelty of the proposed method. From my viewpoint, this paper presents a policy learning method with distributional shift. However, offline policy learning methods with distributional shift is already a popular topic. Moreover, it is not rare to use some DA tricks to improve the transferability of the learned policy across different environments. \n\n2.Related work should distinguish this paper from previous policy learning methods. Their motivations are settings are very similar. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "STh3pHH64uJ", "reviewer": "Reviewer_6jPy", "summary": "This paper studies the offline data-driven model-based optimization problem, where the dataset is static and doesn't allow for collecting new data. The main challenge is to prevent the optimizer from finding an out-of-distribution design. To achieve this, the authors cast this problem as a domain adaption problem and focus on representation learning. To deal with the out-of-distribution solutions, the authors learn an invariant representation which ensures the representation of out-of-distribution designs is uninformative for distinguishing good solutions from bad ones. As a result, the function value of out-of-distribution designs will stay close to the mean of the function values of training data. Consequently, it prevents the optimizer from selecting those designs. The authors also provided analysis, showing that the design found by their methods can compete favorably against the distributions of designs in the training dataset in the worst case. Empirically, the proposed method performs can attain improved designs over the best designs in the training data.", "questions": "See the strengths and weaknesses part.", "limitations": "See the strengths and weaknesses part.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The entire paper is well-structured and easy to follow. In particular, the method of learning an invariant representation to combat the out-of-distribution designs is very well-motivated. In constrat to methods that modify the constraints, the proposed method is compatible with nearly all the off-the-shelf optimizers. \n\n2. The authors also provided many intuitive explanations for their method, which were very helpful for understanding the paper. For example, why will the invariance lead to \"average\" values for those out-of-distribution designs? \n\n3. Though not significant, the authors provide a worst-case analysis for the performance of the method.\n\nWeakness:\n\n4. From my perspective, I feel that the method might be an overkill to the problem. Incorporating representation learning introduces extra but unnecessary complexity to the problem: as a result of the invariant representation, the function value of out-of-distribution designs will be penalized to be the \"average\" function values of the training data. So, why not directly learn an auxiliary function to penalize the function value of those out-of-distribution designs? This should be an easier task than learning an invariant representation.\n\n5. The theoretical analysis is kind of straightforward from the existing literature. For example, similar results have already been shown in offline reinforcement learning (see Theorem 4.4), e.g., Yu et al., 2020. \n\nMinor:\n\n6. For Assumption 4.1, \"there exists a representation \\phi ...\" seems redundant to min_{\\phi, g}. I would  remove \\min_{\\phi, g}.\n\n7. I didn't fully get \"a novel connection with domain adaptation\" (Line 51 - 52). Can you please highlight the \"novel connection\"?\n\n\nOverall, I think this is an interesting work, and it might be interesting for people working on offline reinforcement learning. This paper provides some promise for attacking distribution-shift from the representation learning perspective. \n\n\nYu, Tianhe, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. \"Mopo: Model-based offline policy optimization.\" Advances in Neural Information Processing Systems 33 (2020): 14129-14142.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "A0rWMyJxYGN", "reviewer": "Reviewer_VghW", "summary": "The paper considers the problem of model-based black-box optimization given a static dataset. The main concern is not to report a design that is influenced by model overestimations. The authors propose to formulate this problem as domain adaptation and propose an algorithm IOM that formulates the problem as bi-level optimization and enforces invariance at the representation level by using a specifically designed regularizer. Experimental results show the comparison against other methods for model-based optimization.   \n", "questions": "Please see the previous section.", "limitations": "The authors only stated one limitation that is again very vague in the conclusion. ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I had a hard time reading this paper and this is the primary reason for not being able to grasp the main contributions. Overall, I think that the text can be significantly improved. Even the sections that were supposed to convey intuition were confusing to me. For example, the “Intuition (Line 156)” section did not convey intuition and confused me even more. I did not understand sentences such as:\n* Such an invariance can be enforced when $\\mu_{OPT}$ is sufficiently far away from the training distribution.\n* .. if $\\mu_{OPT}$ is too far away from the training distribution. \n* This would pull the optimizer closer to the training distribution. \n\nThe abstract should do a better job of highlighting the main contributions better. IOM is used in the abstract, but it is unclear what IOM stands for. \n\nNot sure what the main message of Figure 1 is. I think it does not convey intuition well (the last sentence is unclear). \nThings should also be more formal. $\\mu_{OPT}$ and $J(\\mu)$ have the same definition in Eq. 1. It would also be helpful if the distributional shift that the paper talks about is mathematically formalized. \n\nI was also unable to comprehend the theoretical result. If the circle is positive and the star is minimized, then mu_{OPT} is better than mu_{data}. I’m not sure what the utility of such a result is. Again, why do we care about mu_{data} when optimizing, and not only consider the best point/value given the noiseless setting. \n\nOverall, I feel that the entire method is overly complicated. I am not convinced that we need to solve the bi-level optimization problem (which is known to be notoriously hard). While I understand why one might be interested in obtaining “soft” allocation (i.e., distribution over solutions), I do not understand why we need to look at the entire training data distribution and not the best point/value within the given training data. \n\nThe first experiment is also unclear to me. Do you compare it against the best point in the training dataset? How can CMA-ES and REINFORCE be conventional Bayesian Optimization methods?\nI’d suggest looking at more relevant arXiv:2002.09038 and similar model-based (BO) robust works. Another question is how fitting a GP to the training data, defining confidence bounds, and outputting the design that corresponds to the max lower confidence bound performs in this experiment?\n\nSome minor comments:\n– 41: Typo “invariance”\n– 127: Is this Dirac $\\delta$? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2uDbX0aU-HN", "reviewer": "Reviewer_h8mh", "summary": "This paper develops an offline model-based optimization method based on learning invariant representations, a popular approach in domain adaptation. The authors frame the invariant representation learning as a bi-level optimization problem and propose a practical algorithm for solving it. The authors also develop an effective offline hyperparameter selection procedure that's empirically shown to lead to the smallest sub-optimality gap. Experiment results show that the proposed offline MBO method outperforms previous approaches. ", "questions": "1. Figure 1 is confusing. Why is conservatism shown to significantly overestimate the function value in the high-data region? Could the authors provide the reasoning on why this could happen? \n2. In line 92, the authors state \" IOM not only does not require a generative model\" but in line 62-63 \"we instantiate IOM using the $\\chi^2$-discrepancy measure via a least-squares generative adversarial network\". These two statements are conflicting with each other. Could the authors clarify?", "limitations": "The authors addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n1. The proposed offline model-based optimization method based on learning invariant representation is well justified and empirical results show strong performance.\n2. The paper is well-written in general and most parts are easy to follow. \n3. The paper discussed an offline tuning strategy that's critical for real-life applications.\n\n### Weakness\n1. The clarity of the paper could be further improved. Please see my comments in Questions section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2JqcBNngV2U", "reviewer": "Reviewer_1x47", "summary": "This paper proposes a model-based optimization method, which handles the distribution shift challenges through the perspective of domain adaptation. The key idea is to regularize the learned model by treating the training data as the “source domain” and the optimized designs as the “target domain”. The invariance optimizer forces the distribution shift between source and target to be minimal at the representation level, which ensures the out-of-distribution points will not be erroneously overestimated. Overall, the idea of this work is novel to me and the intuition behind is convincing to some extent. Although I didn’t check all details, the proposed bi-level optimization problem and the corresponding practical algorithm appear to be technically sound. ", "questions": "1. The distribution shift in MBO is not clearly defined or illustrated in this paper. In domain adaptation, the source and target domains have different data distributions but share the same feature space. In this paper, the target optimized designs are obtained by optimizing the learned model with respect to the inputs via gradient ascent, similar to adversarial samples. My main concern is whether the adversarial samples equivalent to the case in domain adaptation?  \n2. In section 6.1-Evaluation protocol, how do you test or deploy the produced designs?\n", "limitations": "1. The bi-level optimization objective is not convincing to me. Why do authors formalize IOM training as a bi-level optimization problem with respect to the optimal distribution and the learned model parameters simultaneously? The standard approach of learning a naïve model, which is then optimized via gradient ascent with a invariant regularization term also makes sense to me. Explanations on the necessity of bi-level optimization are welcome.\n2. The author claimed that invariance can effectively handle distribution and conservatism is not necessary. Does it mean IOM is consistently better than conservatism? Any insightful explanations? I suppose that the conclusion needs more examples to support.\n3. The offline tuning strategy needs multiple runs of IOM with a set of hyperparameters, which may be time-consuming. The computational efficiency should be considered. \n4. In section 6.2, more details on oracle turning should be added. Besides, for suboptimality gap in Figure 3, the comparisons with COMs are not fair, since it only considers two naïve case for COMs, more tuning strategies for COMs needs to be evaluated.\n", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength points:\n1. This paper contributes some new ideas. The key idea is formulating offline model-based optimization as domain adaptation, which may have a broad impact.\n2. This paper is clearly written and well organized.\n3. The practical implementation of IOM algorithm and the offline tuning is detailed.\n\nWeaknesses:\nPlease refer to Limitations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept