Model Inversion Networks for Model-Based Optimization
Authors: Aviral Kumar
Sergey Levine
Venue:  (2020)
Keywords: data-driven optimization
model-based optimization

Abstract: In this work, we aim to solve data-driven optimization problems, where the goal is to find an input that maximizes an unknown score function given access to a dataset of input, score pairs. Inputs may lie on extremely thin manifolds in high-dimensional spaces, making the optimization prone to falling-off the manifold. Further, evaluating the unknown function may be expensive, so the algorithm should be able to exploit static, offline data. We propose model inversion networks (MINs) as an approach to solve such problems. Unlike prior work, MINs scale to extremely high-dimensional input spaces and can efficiently leverage offline logged datasets for optimization in both contextual and non-contextual settings. We show that MINs can also be extended to the active setting, commonly studied in prior work, via a simple, novel and effective scheme for active data collection. Our experiments show that MINs act as powerful optimizers on a range of contextual/non-contextual, static/active problems including optimization over images and protein designs and learning from logged bandit feedback.

Under review as a conference paper at ICLR 2020
MODEL INVERSION NETWORKS FOR MODEL-BASED OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, we aim to solve data-driven optimization problems, where the goal is to find an input that maximizes an unknown score function given access to a dataset of input, score pairs. Inputs may lie on extremely thin manifolds in high- dimensional spaces, making the optimization prone to falling-off off the manifold. Further, evaluating the unknown function may be expensive, so the algorithm should be able to exploit static, offline data. We propose model inversion networks (MINs) as an approach to solve such problems. Unlike prior work, MINs scale to extremely high-dimensional input spaces and can efficiently leverage offline logged datasets for optimization in both contextual and non non-contextual settings. We show that MINs can also be extended to the active setting, commonly studied in prior work, via a simple, novel and effective scheme for active data collection. Our experiments show that MINs act as powerful optimizers on a range contextual/non-coneextte static/active problems including optimization over images and protein designs and learning from logged bandit feedback.
INTRODUCTION
Data-driven optimization problems arise in a range of domains: from protein design (Brookes et al. 2019) to automated aircraft design (Hoburg & Abbeel| 2012), from the design of robots (Liao et al. 2019) to the design of neural net architectures (Zoph & Lc 2017) and learning from logged feedback, such as optimizing user preferences in recommender systems. Such problems require optimizing unknown reward or score functions using previously collected data consisting of pairs of inputs and corresponding score values, without direct access to the score function being optimized. This can be especially challenging when valid inputs lie on low-dimensional manifold in the space of all inputs, e.g., the space of valid aircraft designs or valid images. Existing methods to solve such problems often use derivative- active data collection where the unknown function is queried at new inputs. However, when function evaluation involves complex real-world process, such as testing new aircraft design evaluating a new protein, such active methods can be very expensive. On the other hand, in many cases there is considerable prior data existing aireraft and protein designs, and advertisements and user click rates, etc. that could be leveraged to solve the optimization problem.
optimization (Snoek et al.)) Most of these techniques require
In this work, our goal is develop an optimization approach solve such optimization problems that can (1) readily operate on nigh-dimensional inputs comprising a narrow, low-dimensional manifold, such natural images, (2) readily utilize offline static data, and (3) learn with minimal active data collection if needed. We can define this problem setting formally as the optimization problem
arg max fx),
where the function f(x) is unknown, and we have access to a dataset D = {(x1, V1)..... (XN UNN }. where yi denotes the value f(x:). If no further data collection is possible, we call this the data-driven model-based optimization setting. This can also be extended to the contextual setting, where the aim to optimize the expected score function value across a context distribution. That is,
π = arg max Eo If (c)),
(2)
where x* maps contexts e to inputs x. such that the expected score under the context distribution (o(c) is optimized, As before, f(c, x) is unknown and we have access to dataset D ((G,, X1, y1 )

---

Under review as a conference paper at ICLR 2020
where y is the value of x())))).. Such contextual problems with logged datasets have been studied in the context of contextual bandits (Swaminathan & Joachims Joachims et al. 2018).
A simple way to approach these model-based optimization problems is to train proxy function fo (x) or folc, x), with parameters 0, to approximate the true score, using the dataset D. However, directly using (x) in place of the true function f(x) in Equation (1) generally works poorly, because the optimizer will quickly find an input x for which fe (x) outputs an erroneously large value. This issue is especially severe when the inputs lie on a narrow manifold in high-dimensional space, such as the set of natural images (Zhu et al. 2016). The function fx(x) is only valid near the training distribution, and can output erroneously large values when queried at points chosen by the optimizer. Prior work has sought to addresses this issue by using uncertainty estimation and Bayesian models (Snoek et al., 2015) for fo(x,, as well as active data collection (Snoek et al.). However, explicit uncertainty estimation is difficult when the function fo(x) is very complex or when x is high-dimensional.
Instead of learning fo(x), we propose to learn the inverse function, mapping from values y to corresponding inputs x. This inverse mapping is one-to-many, and therefore requires a stochastic mapping, which we can express as fg (v. z) + x. where z is a random variable. We term such models model inversion networks (MiNs). MINs provide us with a number of desirable properties: they can utilize static datasets, handle high- nigh-dimensional input spaces such as images, contextual problems, and can accommodate both static datasets and active data collection. We discuss how to design simple active data collection methods for MINs, leverage advances in deep generative modeling (Goodfellow et al.) Brock et al. 2019), and scale to very high-dimensional input spaces. We experimentally demonstrate MINs in a range of settings, showing that they outperform prior methods on high-dimensional input spaces, perform competitively to Bayesian optimization methods inputs, and substantially outperform prior methods on contextual optimization from logged data (Swaminathan & Joachims
handle
on tasks with active data collection and lower-dimension
RELATED WORK
Bayesian optimization. In this paper, we aim to solve data-driven optimization problems. Most prior work aimed at solving such optimization problems has focused on the active setting. This includes algorithms such as the cross entropy method (CEM) and related derivative-free methods Ru binstein (1996); Rubinstein & Kroese (2004), reward weighted regression Peters & Schaal Bayesian optimization methods based on Gaussian processes Shahriari et al. 6): et al. 2015). and
variants that replace GPs with parametric acquisition function approximators such as Bayesian neural
networks (Snoek et al.) 2015) and latent variable models (Kim et al., 2019 Garnelo et al. 201 8b a). as well as more recent methods such as CbAS (Brookes et al. 2019). These methods require the ability to query the true function f(x at each iteration to iteratively arrive at near-optimal solution. We show in Section 3.3 that MINs can be applied to such an active setting as well, and in our experiments we show that MINs can perform competitively with these prior methods. Additionally, we show that MINS can be applied to the static setting, where these prior methods are not applicable. Furthermore, most conventional BO methods do not scale favourably to high-dimensional input spaces, such as images, while MINs can handle image inputs effectively.
Contextual bandits. Equation Caapuure the class of contextual bandit problems. Prior work on batch contextual bandits has focused on batch learning from bandit feedback (BLBF), where the learner needs to produce the best possible policy that optimizes the score function from logged experience. Existing approaches build on the counterfactual risk minimization (CRM) principle (Swaminathan & Joachims (b) and have been extended to work with deep nets (Joachims al. 2018). In our comparisons, we find that MINs substantially outperform these prior methods in the batch contextual bandit setting.
Deep generative modeling. Recently, deep generative modeling approaches have been very suc- cessful at modelling high-dimensional manifolds such as natural images (Goodfellow et al.) Van Den Oord et al.) Dinh et al. 2016)) speech (van den Oord et al. 2018), text (Yu et al.). alloy compo- sition prediction (Nguyen et al.). etc. MINs combine the strength of such generative models with important algorithmic decisions to solve model-based optimization problems. In our experimental evaluation, we show that these design decisions are important for adapting deep generative models to model-based optimization, and it is difficult to perform effective optimization without them.
2

---

Under review as a conference paper at ICLR 2020
3 MODEL INVERSION NETWORKS
In this section, we describe our model inversion networks (MINs) method, which can perform both active and passive model-based optimization over high-dimennional input spaces.
Problem statement. Our goal is to solve optimization problems of the form x* = arg maxx (x), where the function f(x) is not known, but we must instead use dataset of input-output tuples D = {(x,, 3.) In the contextual setting described in Equation ( each datapoint also associated with context Cç. For clarity, we present method in the non-contextual setting, but the contextual setting can be derived analogously by conditioning all functions on the context. In the active setting, which is most often studied in prior work, the algorithm is allowed to actively query fxx one or more times on each iteration to augment the dataset, while in the static setting, only an initial static dataset is available. The goal is obtain the best possible x* (i.e., the one with highest possible value of f(x*)).
One naive way of solving MBO problems is to learn a proxy score function [x(x), via standard empirical risk minimization. We could then maximize this learned function with respect to x via standard optimization methods. However, naive upplications of such a method would fail for two reasons. First, the proxy function fo (x) may not be accurate outside the samples on which it is trained, and optimization with respect to it may simply lead to values of x for which fo(x) makes the largest mistake in the negative direction. The second problem is more subtle. When x lies on narrow manifold in very high-dimensional space (such as the space of natural images), the optimizer can produce invalid values of x, which result in arbitrary outputs when fed into so (x). Since the shape of this manifold is unknown, it is difficult to constrain the optimizer to prevent this. This second problem is rarely addressed or discussed in prior work, which typically focuses optimization over low-dimensional and compact domains with known bounds.
3.1 OPTIMIZATION VIA INVERSE MAPS
Part of the reason for the brittleness of the naïve approach above is that fo(x) has high- digensinnni input space, making it easy for the optimizer to find inputs x for which the proxy function produces an unreasonable output. Can we instead learn a function with small input space, which implicitly understands the space of valid, in-distribution values for x? The main idea behind our approach is to model an inverse map that produces a value of given a score value y. given by fo : - X. The input to the inverse map is scalar, making it comparatively easy to constrain to valid values, and by directly generating the inputs x, an approximation to the inverse function must implicitly understand which input values are valid. As multiple x values can correspond to the same y. we design fo as a stochastic map that maps a score value along with d dd-dennnsio random vector to a x. fo y x Z + X, where z is distributed according to a prior distribution Po(z).
To define the inverse map objective, Iet the data distribution be denoted (x y). let pD(y) be the marginal over y. and let p(y) be an any distribution defined on y (which could be equal to (y)).. We can train the proxy inverse map fo under distribution p(v) by minimizing the following objective:
C, Ey-p(r) (D(D (x (y) PJ, (xlx))].
(3)
where p,, (xly) is obtained by marginalizing over z, P,. = S2 Po(z) 1 = (z, (2,,z)))]e y)lld,, and D is measure of divergence between the two distributions. Using the Kullback-Leiller divergence leads to maximum likelihood learning, while Jensen-Shannon divergence motivates GAN-style training objective. MINs can be adapted the contextual setting by pass- ing in the context as an input and learning Sø (c)) In standard empirical risk mini- mization. we would choose p(y) to be the data distribution po(y), such that the expectation can be approximated simply by sampling train- ing tuples (x; y1) from the training set. How- ever, as we will discuss in Section 3.3 a more careful choice for p(y) can lead to better perfor- mance. The MIN algorithm is based on training an inverse map. and then using it via the infer- ence procedure in Section 3.2 to infer the that approximately optimizes [x). The structure of the MIN algorithm is shown in Algorithm
Algorithm 1 Generic Algorithm for MINs
1: Input: po(y): distribution of y in D
2: Train inverse map fo y Z + X using objective (Equation 3) with reweighting, active data collection if needed
3: x" + APPROX- PD()))
4: return x"
3

---

Under review as a conference paper at ICLR 2020
3.2 INFERENCE WITH INVERSE MAPS XAPROOX-NNFER
Once the inverse map is trained, the goal of our algorithm is to generate the best possible x", which will maximize the true score function as well as possible under the dataset. Since a score y needs to be provided as input to the inverse map, we must select for which score y query the inverse map to obtain a near-optimal x. One naïve heuristic is to pick the best Umna E "D and produce Xmax fo (ymax) as the output. However, the method should be able to extrapolate beyond the best score seen in the dataset, especially in contextual settings, where a good score may not have been observed for all contexts.
In order to extrapolate as far as possible, while still staying on the valid data manifold, we need to measure the validity of the generated values of x. One way to do this is to measure the agreement between the learned inverse map and an independently trained forward model fø: the values of y for which the generated samples x are predicted to have a score similar to y are likely in-distribution, whereas those where the forward model predicts a very different score may be too far outside the training distribution. Since the latent variable z captures the multiple possible outputs of the one-to- many inverse map, we can further optimize over for given y to find the best, most trustworthy output x. This can be formalized as the following optimization:
V 3" arg max V.S
folfo (2, v)) Aully fo(fo'(z,,)) (2, + A2 belogpo(3 Po())
(4)
This optimization can be motivated as finding an extrapolated score that corresponds to values of x that lie on the valid input manifold, and for which independently trained forward and inverse maps agree. Although this optimization uses an approximate forward map fx(x), we show in our experiments in Section 4 that it produces substantially better results than optimizing with respect to a forward model alone. The inverse map substantially constraints the search space, requiring an optimization over 1 1-dimensional and (relatively) www-dimensional z, rather than the full space of inputs. This scheme can be viewed as a special (deterministic) case of a probabilistic optimization procedure described in Appendix A
3.3 REWEIGHTING THE TRAINING DISTRIBUTION
naive implementation of the training objective in Equation (3) samples y from the data distribution pD(y), However, as we are most interested in the inverse map's 's predictions for high values of it is much less important for the inverse map to predict accurate x values for values of y that are far from the optimum. We could consider increasing the weights datapoints with larger values of y. In the extreme case, we could train only on the best datapoint- - either the single datapoint with the largest y or, in the contextual case, the datapoint with the largest y for each context. More generally, we can define the optimal y distribution p" (v), which is simply the delta function centered on the best y. p" (v) = (y). in the deterministic case. If we instead assume that the observed scores have additive noise (i.e., we observe f(x) + ~ A\, then (1) would be distribution centered around the optimal y. Of course, training on (vy is not practical, since it heavily down-weights most of the training data, leading to very high-variance training objective, and is not even known in general, since the optimal data point is likely not in our training set. In this section, we will propose better choice for p(y) that trades off the variance due to an overly peaked training distribution and the bias due to training on the "wrong" distribution (i.e., anything other than p" p²()).
We can train under a distribution other than the empirical distribution by using importance sampling, p(y.) such that sample from pD and assign importance weight, given by w, "p(a,'' to each datapoint (x,, y:), where p(y:) is our desired distribution. The reweighted objective is given by (D) = D E. D(x;, fo 6, estimate, with respect to the reweighted objective without sampling error under y drawn from p (2), we obtain the following result: (Proof in Appendix B)
(yx)). fyx)). By bounding the variance and the bias of the gradient of
Theorem 3.1 ((nnormall) Bias + variance bound in MINs). Let L(p²) be the objective under p" (v) without sampling error: L(p") Eyyup (w)) ((p(x(y). (- Let Ny be the number of datapoints with the particular y value observed in D. For some constants C1, C2, Oz, with high confidence,
+ C2 de(p) (pD) + C3 -DV(D"'.. .p)²
E (
< CyEyp(i)
(D)
Theorem 3.1
suggests a tradeoff between being close to the optimal distribution p² (v) and reducing variance by covering the full data distribution pD. We observe that the distribution p(y) that minimizes
4

---

Under review as a conference paper at ICLR 2020
N, the RHS bound in Theorem 3.1 has the following form: p(y) of N, -(p((v)) (v)), where g(p") is linear function of p" (y) that ensures that the distributions p and p* are close. Theoretically, g()) is an increasing, piece-wise linear function of o. We can interpret the expression for p(y) as a product of two likelihoods - the optimality of a particular y value and the likelihood of a particular y not being rare in D. We empirically choose an exponential parameteric form for this function, which we describe in Section 3.5 This upweights the samples with higher scores, reduces the weight on rare y-values (i.e., those with low Ny), while preventing the weight on common y-values from growing, N,+K saturates to 1 for large Ny. This is consistent with our intuition: we would like to upweight datapoints with high y-values, provided the number of samples at those values is not too low. Of course, for continuous-valuee scores, we rarely see the same score twice. Therefore, we bin the y-values into discrete bins for the purpose of weighting, as we discuss in Section 3.5
since N₂
3.4 ACTIVE DATA COLLECTION VIA RANDOMIZED LABELING
While the passive setting requires care in finding the best value of for the inverse map, the active setting presents a different challenge: choosing a new query point x at each iteration to augment the dataset D and make it possible to find the best possible optimum. Prior work on bandits and Bayesian optimization often uses Thompson sampling (TS) (Russo & Van Roy 2016; Russo et al., 2018 Srinivas et a the data-collection strategy. TS maintains a posterior distribution over functions p(aa)Daa). At each iteration, it samples a function from this distribution and queries the point x that greedily minimizes this function. TS offers an appealing query mechanism, since it achieves sub-linear Bayesian regret (defined as the expected cumulative difference between the value of the optimal input and the selected input), given by O ((VT), where T' is the number of queries.
Maintaining posterior over high-dimensional parametric functions is generally intractable. However, we can devise a scheme to approximate Thompson sampling with MINs. To derive this method, first note that sampling fa from the posterior is equivalent sampling (x. 3) pairs consistent with fr - given sufficiently many (x, 3) pairs, there is a unique smooth function fo that satisfies y; = ft (x.). For example, we can infer quadratic function exactly from three points. For a more formal description, we refer readers to the notion of Eluder dimension (Russo & Van Roy). Thus, instead of maintaining intractable beliefs over the function, we identify a function by the samples it generates, and define way to sample synthetic (x, v) points such that they implicitly define unique function sample from the posterior.
To apply this idea to MINs, train the inverse map fø. at each iteration with augmented dataset Dj= = D, US, where S, = {(x) új) )5 is dataset of synthetically generated input-score pairs corresponding to unseen values in D.. Training fo on D/ corresponds to training fi.' to be an approximate inverse map for a function f, sampled from p(ʃi (D1:), as the synthetically generated samples S, implicitly induce model of ft. We can then approximate Thompson sampling by obtaining x from fo. labeling it via the true function, and adding it to D, to produce D.+1- Pseudocode for this method, which we call "randomized labeling," presented in Algorithm ? In Appendix C we further derive ((VT) regret guarantees under mild assumptions. Implementation- wise, this method is simple, does not require estimating explicit uncertainty, and works with arbitrary function classes, including deep neural networks.
Algorithm 2 Active Data Collection with Model Inversion Networks via Randomized Labeling
1: Initialize inverse map. - x Z + X, dataset Do 0.
2: for step in (0. ..., T-1) do
3: Sample synthetic samples S. = ()),,, corresponding to unseen data points y, (by randomly pairing noisy observed x, values with unobserved y values.)
4: Train inverse map f:
DJ = D. U St. using reweighting described in Section 3.3
5: Query function f at X = fr (maxp: y)
6: Observe outcome: (xx, f (xx)) and update D.+1 De U (xx, f(x))
7: end for
3.5 PRACTICAL IMPLEMENTATION OF MINS
In this section, we describe our instantiation of MINs for high-dimensional inputs with deep neural network models. GANs (Goodfellow et al.) have been successfully used to model the manifold of
5

---

Under review as a conference paper at ICLR 2020
high-dimensional inputs, without the need for explicit density modelling and are known to produce
more realistic samples than other models such as VAEs (Kingma & Welling 2013) or Flows (Dinh] et al. 2016). The inverse map in MINs needs to model the manifold of valid x thus making GANs suitable choice. We can instantiate our inverse map with a GAN by choosing D in Equation to be the Jensen-Shannon divergence measure. Since we generate x conditioned on y. the discriminator is parameterized as Disc(xly), and trained to output for valid (x, y) pair (i.e., where f(x) and
x comes from the data) and otherwise. Thus, we optimize the following objective:
min max Lp (D) = By-p(s) (x-) [log Disc(r y) + (x- [log(1 1 Disc(fø (z. v) /)]] Disc
This model is similar to a conditional GAN (cGAN), which has been used in the context of modeling
distribution of x conditioned on discrete-valued label (Mirza & Osindero 2014). As discussed
in Section 3.3 we additionally reweight the data distribution using importance sampling. To that
end, we discretize the space y into B discrete bins bj.... .bg and, following Section [.3] weight
No. each bin b, according to p(b)) No, exp [b.-8"l) where No, is the number of datapoints
in the bin, y² is the maximum score observed, and is hyperparameter. (After discretization,
using notation from Section 3.3 for any that lies in bin b, (")) : pb(b) - =exp ( (b and
No p(y) = (bb) & N exp
b-v"
Experimental details are provided in Appendix C.4.
In the active setting, we perform active data collection using the synthetic relabelling algorithm described in Section 3.4 In practice, we train two copies of fz The first, which we call the exploration model Teppl= is trained with data augmented via symthetically generated samples (i.c., D(). The other copy, called the exploitation model fexplete is trained on only real samples (i.e., D,). This improves stability during training, while still performing data collection as dictated by Algorithm 2 To generate the augmented dataset D' in practice, we sample values from p" (y) (the distribution over high- scoring yys observed in Dz), and add positive-valued noise, thus making the augmented y values higher than those in the dataset which promotes exploration. The corresponding inputs x are simply sampled from the dataset Dj or uniformly sampled from the bounded input domain when provided in the problem statement. (for example, benchmark function optimization) After training. we infer best possible x* from the trained model using the inference procedure described in Section 3.2 In the active setting, the inference procedure is applied on fexploie the inverse map that is trained only real data points.
4 EXPERIMENTAL EVALUATION
The goal of our empirical evaluation is to answer the following questions. (1) Can MINs successfully solve optimization problems of the form shown in Equations [ and 2 in static settings and active settings, better than or comparable to prior methods? (2) Can MINs generalize to high dimensional spaces, where valid inputs x lie on a lower-dimensional manifold, such as the space of natural images? (3) Is reweighting the data distribution important for effective data-driven model-based optimization? (4) Does our proposed inference procedure effectively discover valid inputs x with better values than any value seen in the dataset? (5) Does randomized labeling help in active data collection?
4.1 DATA-DRIVEN OPTIMIZATION WITH STATIC DATASETS
We first study the data-driven model-based optimization setting. This requires generating points that achieve a better function value than any point in the training set or, in the contextual setting, better than the policy that generated the dataset for each context. We evaluate our method on a batch contextual bandit task proposed in prior work (Joachims et al. 2018) and on a high-dimension contextual image optimization task. We also evaluate method on several non-contextual tasks that require optimizing over high-dimensions image inputs to evaluate a semantic score function, including hand-writd written characters and real-world photographs.
Batch contextual bandits. We first study the contextual optimization problem described in Equa- tion 2 The goal is to learn a policy, purely from static data, that predicts the correct bandit arm for each context c. such that the policy achieves a high overall score f(c, π(c)) on average across contexts drawn from distribution po(c). We follow the protocol set out by Joachims al. (2018).

---

Under review as a conference paper at ICLR 2020
Dataset & Type
BanditNet
BanditNet"
MIN w/ 94.2
MIN (Ours)
MINs w/o R
MNIST (49% comr.
36.42 0.6
95.0 + 0.16
95.0 = 0.21
MNIST (Uniform)
9.94 + 0.0
92.21 + 0.22
93.67 + 0.51
92.8 - 0.01
CIFAR- 10 (49% corr.)
42.13 - 2.35
87.0
91.35 0.87
92.21 1.0
89.02 0.05
CIFAR- 10 (Uniform)
14.43 + 1.43
76.31 + 0.40
77.12 + 0.54
74.87 = 0.12
Table 1: Test accuracy on MNIST and CIFAR-I0 with 50k bandit feedback training examples. BanditNet" is the result from Joachims et al. (2018), while the BanditNct column is our implementation; we were unable to replicate the performance from prior work (dctails in Appendix (D. MINs outperform both Ben and BanditNet", both with and without the inference procedure in Section 3.2 MINs w/o reweighting perform at par
with full MINs on MNIST, and slightly worse on CIFAR 10, while still outperforming the baseline.
MIN MIN-I MIN-R F
MIN
MIN-I MIN-R
F
MIN MIN-I
MIN-R F
$
3
2
3
3
3
E
1
7
7
3
3
3
3
x
2
3
8
8
7
8
8
2
3
3
3
8
7
6
9
276.3
341.3
242.40 453.2
255.4
240.8
333,4
515.7
2.32
1.12
1.53
1.42
(a) Thickest stroke
(b) Thickest digit (3)
(c) Most number of blobs (8)
Figure 1: Results for non-contextual static dataset optimization on MNIST: (a) and (b): Stroke width optimization,
and (c): Maximization of disconnected black pixel blobs. From left to right: MINs, MINs w/o Inference (Section 3.2 which sample from the inverse map conditioned on the highest seen value of y, MINs w/o Reweighting (Section 3.3) and direct optimization of a forward model, which starts with a random dataset image and optimizes it for the highest score based on the forward model. Observe that MINs can produce thickest characters which resemble valid digits. Optimizing the forward function often turns non-digit pixels on, thus going off the valid manifold. Both the reweighting and inference procedure results. Quantitative results are provided in Appendix [33] Different rows (for F) are obtained by optimizing from different initial points, Scores are listed beneath cach figure. The larger score the better, provided the solution x is the image of valid digit.
important for good
which evaluates contextual bandit policies trained on static dataset for a simulated classification tasks. The data constructed by selecting images from the (MNIST/CIFAR) dataset as the context e, a random label as the input x, and a binary indicator indicating whether or not the label is correct as the score y. Multiple schemes can be used for selecting random labels for generating the dataset, and we evaluate on two such schemes, as described below. We report the average score set of new contexts, which equal to the average 0-1 accuracy of the learned model on held out test set of images (contexts). We compare our method to previously proposed techniques, including the BanditNet model proposed by DJacchis et al. (2018) on the MÑIST and CIFAR- 10 (Krizhessyy) 2009) datasets. Note that this task is different from regular classification, in that the observed feedback (,,,,x,,;)) pairs) is partial, i.e. we do not observe the correct label for each context (image) c, but only whether or not the label in the training tuple is correct or not. We evaluate on two datasets: (1) data generated by selecting random labels x; for each context e; and (2) data where the correct label is used 49% of the time, which matches the protocol in prior work (Joachims et al., 2018). We compare to BanditNet (Joachims et al.) 2018) on identical dataset splits. We report the average 0-1 test accuracy for all methods in Table I The results show that MINs drastically outperform BanditNet on both MNIST and CIFAR- 10 datasets, indicating that MINs can successfully perform contextual model-based optimization in the static (data-driven) setting. The results also show that utilizing the inference procedure in Section 3 2 produces improvement of about 1.5% and 1.0% in test-accuracy on MNIST and CIFAR- 10, respectively.
Character stroke width optimization. In the next experiment, we study how well MINs optimize over high- -igh-dimensional inputs, where valid inputs lie on diver-dimensionall manifold. We constructed an image optimization task out of the MNIST (LeCun & Cortes 2010) dataset. The goal is optimize directly over the image pixels, to produce images with the thickest stroke width, such that the image corresponds either (a) any valid character or (b) valid instance of particular character class. A
7

---

Under review as a conference paper at ICLR 2020
<15
15-20
20-25
25-30
9.8
13.0
17.0
18.0
22.0
23.0
29.0
26.0
14.0
14.0
19.0
16.0
21.0
24.0
26.0
25.0
12.6
13.2
12.2
16.4
27.5 1
23.
24.8
29.00
(a) Optimized x (trained on years)
(b) Optimized x (trained on years)
Figure 2: MIN optimization to obtain the youngest faces when trained on faces older than 15 (left) and older than 25 (right). Generated faces (bottom) are obtained via inference in the inverse map at different points during model training. Real faces of varying ages (including ages lower than those used to train the model) are shown the top rows. We overlay the actual age (negative and the age obtained from subjective user rankings on the generated faces. The score function being optimized (maximized) in this case is the negative age of the face.
score function) for each face on the real images,
successful algorithm will produce the thickest character that is still recognizable. In Figure ! we observe that MINs generate images x that maximize the respective score functions in each case. We also evaluate on a harder task where the goal is to maximize the number of disconnected blobs of black pixels in an image of a digit. For comparison, we evaluate method that directly optimizes the image pixels with respect the manifold of valid characters. We also compare to MINs without the reweighting scheme and the inference procedure, where y is the maximum possible y in the dataset to demonstrate the benefits of these two aspects.
forward model, of the form fo(x). In this case, the solutions are far off
Table 2:
Quantitative score- values for Youngest Face Opti- mization Task (larger the better)
Semantic image optimization. The goal in these tasks is to quan- tify the ability of MINs to optimize high-level properties that require semantic understanding of images. We consider MBO tasks on the faces (Rothe et al., 2015 2016) dataset, where the func- tion f (x) is the negative of the age of the person in the image. Hence, images with younger people have higher scores.
Task
MIN
MIN (best)
2 15
-13.6
-12.2
We construct two versions of this task: one where the training data consists of all faces older than 15 years, and the other where the model is trained on all faces older than 25 years.The ensures that our model cannot simply copy the youngest face. To obtain ground truth scores for the generated faces, we use subjective judgement from human participants. We perform a study with 13 users. Each user was asked to answer a set of 35 binary-choice questions each asking the user to pick the older image of the two provided alternatives. We then fit an age function to this set of binary preferences, analogously to Christianoo
2 25
-26.2
-23.9
al. (017).
Figure 2 shows the images produced by MINs. For comparison, we also present some sample of images from the dataset partitioned by the ground truth score. We find that the most likely age for optimal images produced by training MINs on images of people 15 years or older was 13.6 years, with the best image having an age of 12.2. The model trained on ages 25 and above produced more mixed results, with an average age of 26.2, and minimum age of 23.9. We report these results in Table 2 This task is exceptionally difficult, since the model must extrapolate outside of the ages seen in the training set, picking up on patterns in the images that can be used to produce faces that appear younger than any face that the model had seen, while avoiding unrealistic images.
We also conducted experiments on contextual image optimization with MINs. We studied contextual optimization over writtwritten digits to maximize stroke width, using either the character category as the context c, or the top one-fourth or top half of the image. In the latter case, MINs must learn to complete the image while maximizing for the stroke width. In the case of class-conding conditiond optimization, MINs attain average score over the classes of
Table 3: Quantitative score values for MNIST inpainting (contextual)
Mask
MIN
Dataset
mask A
223.57
149.0
mask B
234.32
149.0
8

---

Under review as a conference paper at ICLR 2020
237.6, while the dataset average is 149.0. In the case where the context is the top half quarter of the image, MINs obtain average scores of 223.57 and 234.32, respectively, while the dataset average is 149.0 for both tasks. We report these results in Table [ We also conducted a contextual optimization experiment on faces from the Celeb-A dataset, with some example images shown in Figure 3 The context corresponds to the choice for the attributes brown hair, black hair, hangs, or moustache. The optimization score is given by the sum of the attributes wavy hair, eyeglasses, smiling, and no beard. Qualitatively, we can see that MINs successfully optimize the score while obeying the target context, though evaluating the true score is impossible without subjective judgement on this task. We discuss these experiments in more detail in Appendix D.I
4.2 OPTIMIZATION WITH ACTIVE DATA COLLECTION
In the active MBO setting, MINs must select which new datapoints to query to improve their estimate of the optimal input. In this setting, we compare prior model-based optimization methods, and evaluate the exploration technique described in Section 3.4
Brown or Black Hair
Global optimization on benchmark functions. We first com- pare MINs to prior work in Bayesian optimization on standard benchmark problems (DNGO) (Snoek et al. 2015): the 2D Branin function, and the 6D Hartmann function. As shown in Table _ MINs reach within +-0.1 units of the global minimum (minimiza- tion is performed here, instead of maximization). performing comparably with commonly used Bayesian optimization meth- ods based Gaussian processes. We do not expect MINs to be as efficient as GP-based methods, since MINs rely on train- ing parametric neural networks with many parameters, which is less efficient than GPs on low-dimensional tasks. Exact Gaussian processes and adaptive Bayesian linear regression (Snock et al., 2015) outperform MINs in terms of optimization precision and the number of samples queried, but MINs achieve comparable performance with about 4x more samples. We also report the performance of MINs without the random labeling exploration method, instead selecting the next query point by greedily maximizing the current model with some additive noise. We find that the random relabeling method produces substantially better results than the greedy data collection approach, indicating the importance of effective exploration methods for MINs.
Both Brown/Black Hair
Figure 3: Optimized x produced from contextual training Celeb-
Context (brown hair, black hair, bangs, moustache and f(xx = & (wavy hair, eyeglasses, smiling, no beard). We show the produced for two contexts. The model op- timizes score for both observed con- texts such as brown or black hair and extrapolates to unobserved contexts such brown and black hair.
Function
Spearmint 0.398 + 0.0
DNGO
MIN
MIN + greedy 0.4 + 0.05(800)
Branin (0.398)
0.398 + 0.0
0.398 + 0.02 -3.315 0.05(
Hartmann6 (-3.322)
-3.3166 + 0.02
-3.319 + 0.00
-3.092 + O. 0.12(1)00)
Table Active MBO benchmark functions. The prior methods converge within 200 iterations. MINs require more iterations on some of the tasks, in which case we indicate the number of iterations in brackets. MINs reach similar final performance, and typically require 1-4x as much data as efficient GP-based algorithms.
Protein fluorescence maximization. In the next experiment, we study -igh-dimensional active MBO task, previously studied by [Brookes et al. 2019)). This task requires optimizing over protein designs by selecting variable length sequences of codons, where each codon can take on one of 20 values. In order to model discrete values, we use a Gumbel-softmax GAN also previously employed in Gupta & Zou 2018). and as a baseline in (Brookes al.) 2019). For backpropagation, we choose a temperature = 0.75 for the Gumbel-softmax operation. This is also mentioned in Appendix D The aim in this task is to produce a protein with maximum fluorescence. Each algorithm is provided with a starting dataset, and then allowed a identical, limited number of score function queries. For each query made by an algorithm, it receives a score value from an oracle. We use the trained oracles released by [Brookes et al. (2019). These oracles are separately trained forward models, and can potentially be inaccurate, especially for datapoints not observed in the starting static dataset. We compare to CbAS (Brookes et al. (219) and other baselines, including CEM (Cross Entropy Method), RWR (Reward Weighted Regression) and a method that uses a forward model GB
al. 2018 reported by Brookes al, (211). For evaluation, we report
9

---

Under review as a conference paper at ICLR 2020
the groundtruth score of the output of optimization (max), and the -0th-peeeeetlle groundtruth score of all the samples produced via sampling (this is without inference in the MIN case) so as to be comparable to Broooes et al. (2019). In Table [] we show that MINs comparable to the best performing method on this task, and produce samples with the highest score among all the methods considered.
These results suggest that MINs can perform competitively with previously proposed model-based optimization methods the active setting, reaching comparable or better performance when compared both to Bayesian optimization methods and previously proposed methods for a higher-dimensional protein design task.
5 DISCUSSION
In this work, we presented novel approach towards model-based optimization (MBO). Instead of learning a proxy forward func- tion fo(x) from inputs x to scores y. MINs learn a stochastic inverse mapping from scores out-of-distribution inputs and can optimize over high dimensional values where valid inputs lie narrow manifold. By using simple and principled design decisions, such as re-weighting the data distribution, MINs can perform effective model-based op- timization even from static, previously collected datasets in the data-driven setting without the need for active data collection. We also described ways to perform active data collection if needed, Our experiments showed that MINs are capable of solving MBO optimization tasks in both contextual and non-contextual settings, and are effective over highly semantic score functions such as age of the person in an image.
Method
Max
50%ile
MIN (Ours)
3,42
3.24
MIN -R
3.37
3.28
inputs. MINs
resistent to
CbAS
3.36
3.28
RWR
3.00
2.97
CEM-PI
2.92
~ 2.9
GB
~v 3.25
~325
Table 5: Protein design results, with maximum fluorescence and the 50 percentile out of 100 samples. Prior method results from [rookee et al. (2019). MINs perform com- parably to CbAS. MINs without reweighting (MIN-R) lead to more consistent sample quality (higher 50%ile score), while MINs with reweighting can produce the highest scoring sample.
Prior work has usually considered MBO in the active or "on- policy" setting, where the algorithm actively queries data it learns. In this work, we introduced the data-driven MBO problem statement and devised method to perform optimization in such scenarios, This is important in settings where data collection is expensive and where abundant datasets exist, for example, protein design, aircraft design and drug design. Further, MINs define family of algorithms that show promising results on MBO problems extremely large input spaces.
While MINs scale high-dimensiones tasks such as model- optimization over images, and are performant in both contextual and non-contextual settings, we believe there are a number of interesting open questions for future work. The interaction between active data collection and reweighting should be investigated in more detail, and poses interesting consequences for MBO, bandits and reinforcement learning. Better and more principled inference procedures are also direction for future work. Another avenue is to study various choices of training objectives in MIN optimization.
REFERENCES
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference Learning Representations, 2019. URL
https://cpenxeviees net/
David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. In Proceedings of the 36th International Conference Machine Learning. PMLR, 2019.
mlr. prwps http//99//booooo
9a. html
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NIPS, 2017.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. CoRR, abs/ 0abs/1605.08880 2016. URL http:: http::/dbbiy /ttp://bbppppp. www-trieeeee de/db, ab/journals/ corr/corri 605. html 1 h #DinhsB16
10

---

Under review as a conference paper at ICLR 2020
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional neural processes. In Proceedings of the 35th International Conference Machine Learning. PMLR, 2018a.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Teh. Neural processes. CoRR, abs/1807.01622, 2018b. URL http:
7/a paaaxiv..oom w/aaxxiv.rrg///// www.......g.b///00000 01622
Rafael
David Duvenaud, José Miguel Hernández-Lobato, Jorge Aguilera- Iparraguirre, Timothy D. Hirzel, Ryan Adams, and Alán Aspuru-Guzik, Automatic chemical design using data-driven continuous representation of molecules, In ACS central science, 2018.
Ian J. Goodfellow, Jean Pouget-Abadie. Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NIPS 14.
Anvita Gupta and James Zou. Feedback gan (fbgan) for dna: novel feedback-loop architecture for optimizing protein functions. ArXiv, abs// 804.01694, 2018.
Warren Hoburg and Pieter Abbeel. Geometric programming for aircraft design optimization. vol- ume 52, 04 2012. ISBN 978- 978-1660086-937-1 doi: 10.2514/..0122-166
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizztionn with gumbel-softmax,
CoRR, abs/1611.01144 2016. URL http://dblp. www-ttiiieiieeeeee detpi/errddd////eee /b/onnaa//u
corr/corr 611.6116611 htm (iEmm##Jangee #JangGP16
Thorsten Joachims, Adith Swaminathan, and Maarten de Rijke. Deep learning with logged bandit
feedback. International Conference on Learning Representations, 2018. URL https://
openreview. net.//ewweeetteeee forte//oummmmm 2id-unaddaaaa....
_xXD
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol
Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference Learning Representations, 2019,
SKE6P
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2013. URL http:// //
arxiv.org/abe.. www.or///bbb...oee a112 6114 cite arxiv:1312.6114
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http:://anm
Lecun. cowm/exbbmmiist....mmommm
Thomas Liao, Grant Wang, Brian Yang, Rene Lee, Kristofer Pister, Sergey Levine, and Roberto Calandra. Data-efficient learning of morphology and controller for microrobot. In 2019 JEEE International Conference on Robotics and Automation, 2019. URL htttps:/aaxxiv.or
abs/ 0bs/1195.00333 01334
Erik Linder-Norén. PyTorch GAN. URL PyTorch-GAN
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Alberto Maria Metelli, Matteo Papini, Francesco Faccio, and Marcello Restelli. Policy optimization via importance sampling. NIPS' 18, 2018. URL http::/t.aaaaa.a. hcwp::///lllcmmm...mcmmmmmm wtt.:aa.......cc.aaaao.oo fim?
1d=3327345.332244
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets, 2014. URL http:
a //axxiv. org. abs//1111 1784 cite 141.::411.1176
Phuoe Nguyen, Truyen Tran, Sunil Gupta, Santu Rana, Matthew Barnett, and Svetha Venkatesh. Incomplete Conditional Density Estimation for Fast Materials Discovery, pp. 549-557. doi: 191117779988111197777777
URL htlhhttss:/// :/epubs.///ppb.../ /epubs. s siam. www.:ogg//oooooo aatbooi/abssssaa 1133//
9781 6 975673.6

---

Under review as a conference paper at ICLR 2020
Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational discriminator bottleneck: Improving imitation learning, inverse RL, and GANs by constraining information flow. In International Conference on Learning Representations, 2019. URLhttps
/ //openreviews new. net 1 forum?idd-yxxxx3333
Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th International Conference on Machine Learning, ICML '07.
Rasmus Rothe, Radu Timofte, and Lue Van Gool. Dex:
Deep expectation of apparent age from single image. In IEEE International Conference on Computer Vision Workshops (ICCVW), December 2015.
Rasmus Rothe, Radu Timofte, and Lue Van Gool. Deep expectation of real and apparent age from a single image without facial landmarks. International Journal of Computer Vision (IJCV), July 2016.
Reuven Y. Rubinstein. Optimization of computer simulation models with rare events. European Journal of Operations Research, 99:89-118 12, 1996.
Reuven Y. Rubinstein and Dirk P. Kroese. The Cross Entropy Method: A Unified Approach To Com- binatorial Optimization, Monte-carlo Simulation (Information Science and Statistics). Springer- Verlag, Berlin, Heidelberg, 2004, ISBN 038721240X
Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. In Advances in Neural Information Processing Systems 26.
Daniel Russo and Benjamin Van Roy. An information-tthorrcii analysis of thompson sampling. J. Mach. Learn. Res., 17(1) Jamuary 2016. ISSN 1532-4435, hotthtt://// //cl::///l... acm.
org/citation cfm wef.:mdd=244666555 3007021
Daniel J. Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. A tutorial on thompson sampling. Found. Trends Mach. Learn., 11(1): July 2018. ISSN 1935-8237. doi: 10.1561/2200000000. URL_hppp..//aa1..og0000111611//1
Noveen Sachdeva. BanditNet.
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104:148-111 2016.
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In Proceedings of the 25th International Conference on Neural Information Processing Systems Volume NIPS' 12. http://p//// //tp://ddlae acm. org/ citation. c ffm?
1d=2999325. 2999464
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In Proceedings of the 32nd International Conference Machine Learning. PMLR, 2015.
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process opti- mization in the bandit setting: No regret and experimental design. In Proceedings of the 27th International Conference International Conference on Machine Learning, ICML 10.
Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. In Proceedings of the 32Nd International Conference International Conference on Machine Learning Volume 37, ICML215, a.
Adith Swaminathan and Thorsten Joachims. The nelf-noomalized estimator for counterfactual learning. In Proceedings of the 28th International Conference on Neural Information Processing Systems- Volume 2, NIPS'15. 15, b.
12

---

Under review as a conference paper at ICLR 2020
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of the 33rd Interaatinnal Conference on International Conference on Machine Learning- Volume 48, ICML'16.
Aaron van den Oord, Yazhe Li, Igor Babuschkin, and et.al. Parallel WaveNet: Fast high-fidelity speech synthesis. In Proceedings of the 35th International Conference on Machine Learning. PMLR, 2018.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the Thirty-Firsl AAAI Conference on Artificial Intelligence, 1AAI'17.
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros. Generative visual manipula- tion on the natural image manifold. In Proceedings of European Conference Computer Vision (ECCV), 2016.
Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. 2017. URL https://arxiv wttp/b/////...00/////
13

---

Under review as a conference paper at ICLR 2020
A PROBABILISTIC INTERPRETATION OF SECTION 3.2
In this section, we show that the inference scheme described Equation Section 3.2 emerges as deterministic relaxation of the probabilistic inference scheme described below. We re-iterate that in Section 3.2 singleton x² is the output of optimization, however the procedure can be motivated from the perspective of the following probabilistic inference scheme.
Let (X denote a stochastic inverse map, and let Py (ylaz) be a probabilistic forward map. Consider the following optimization problem:
arg Exx- (8/xs) [+]
v.P
such that H(ū Σ 61,
E2
where (xly)) is the probability distribution induced by the learned inverse map (in case, this corresponds to the distribution of (v, z) induced due to randomness in Po(-)), Py (xly) is the learned forward map. is Shannon entropy, and is KL-divergence measure between two distributions. In Equation 4 maximization is carried out over the input y to the invere-mma,, and the input x which is captured in ş in the above optimization problem, i.e. maximization over 2 in Equation is equivalent to choosing P subject the choice of singleton/ Dirac-delta p. The Lagrangian is given by:
Cly.p: Ex-pptlo) [0] A (Bx--pt [logpr(0(x)] E1) A2 D((x()))) pe (x B)))
In order to derive Equation 4 we restrict P to the Dirac-delta distribution generated by querying the learned inverse map lo specific value of z. Now note that the first term in the Lagrangian corresponds to maximizing the "reconstructed" ý similarly to the first term in Equation 4 If py is
assumed to be a Gaussian random variable with a fixed variance, then log py [ū)x x) = -11 (x))
where μ the mean of the probabílistic forward map. With deterministic forward maps, we make assumption that xxx) = y (the queried value of y), which gives us the second term from Equation
Finally, order to obtain the log Pu()) term, note that, D(p(x(y). (x S D(6,(().o(())
log Po(2) (by the data processing inequality for KL-diiergeecce. Hence, constraining log (=)
instead of the true divergence gives us a lower bound on L. Maximizing this lower bound (which the same as Equation hence also maximizes the true Lagrangian L.
B BIAS- VARIANC TRADEOFF DURING MIN TRAINING
In this section, we provide details on the bias-variance tradeoff that arises in MIN training. Our analysis is primarily based on analysing the bias and variance in the f2 norm the gradient in two cases if we had access to infinte samples of the distribution over optimal ys, p (y) (this is Dirac-delta distribution when function f(x) evaluations are deterministic, and a distribution with non-zero variance when the function evaluations stochastic are corrupted by noise). Let
INN, f denote the empirical objective that the inverse map is trained with. We first analyze the variance of the gradient estimator in Lemma [.22] order to analyse this, we will need the expression for variance of the importance sampling estimator, which is captured in the following Lemma.
(D) D Σy,opo()) Po(3)) Py
p(g))
Lemma B.1 (Variance of IS (Metelli et al.) 2018)). Let P and Q be two probability measures on the space (X, F) such that d2(P) (PIIQ) Co. Let XI.... XN be N randomly drawn samples from Q, and f X > TK is a cniformly-boundee function. Then for any E (0, 1), with probability atleast 1 8,
N N Σ
(P/o(a))((s))) + 11 (1 (1-8)d2(PI) (PIIQ)
E Ba-r(((>)
8N
Equipped with Lemma B.I we are ready to show the variance in the gradient due to reweighting to a distribution for which only few datapoints observed.
14

---

Under review as a conference paper at ICLR 2020
Lemma B.2 (Gradient Variance Bound for MINs). Let the inverse map be given by fø Let Ny denote the number of datapoints observed 'D with score equal y, and let (p(D) be as defined above. Let Lp(pD) = (D),, where the expectation is computed with respect to the dataset DD Assume that & Dxx (y))ll2 S L and var[ (x,, f1()] S o². Then, there exist some constants C₁ C2 such that with a confidence at least 1 6.
0 log !
+ C, (1-11-00 -))d((plPp)) Ny
B |ooL voccppol)-ooopool
C1By-pi))
σ No
Proof. We first bound the range in which the random variable (D) can take values function of number of samples observed for each All the steps follow with high probability, i.e. with probability greater than 6,
1N Σ
plys) (1)))
VoCx(D) V
Σ (1)
D(x f (ws)
DD'ol
NNss
var(D(a, v)) (log +
[ +V
Bxx (ixps())
DDD
N.N,
\j>pp((u)
[D(x + Excxpp((es) rppr)
var( (Drr v)) (log 8 N₂
8) d2 (p(())) (p(v))(p(()))
(D(())
8. En, ED No
(5)
where d2(pl) is the exponentiated Renyi-divergence between the two distributions p and q. i.c. (2((())l)())) = f, g(v) each inner term in the sum corresponding to yj and then bounding the variance due importance sampling ys finally using concentration bounds on variance of importance sampling using Lemma
p(v)
dy. The first step follows by applying Hoeffding's inequality on
a(v)
Thus, the gradient can fluctuate in the entire range of values as defined above with high probability. Thus, with high probability, atleast 1 - 6,
[2 log
+ Cc -8)d2(P((PD)
|VoL VoC,(PD) S (C
(6)
Ng
8E No
The next step is to bound the bias in the gradient that arises due to training on a different distribution than the distribution of optimal ys. p" (v). This can be written as follows:
|Ey=p' (v) (xx=p)())) (xx,
r)]] [D(x, (Ex [D(x, e)llII Drv(p.p").2 L.
where Drv is the total variation divergence between distributions p and p", and I. is constant that depends on the maximum magnitude of the divergence measure D. Combining Lemma B.2 and the above result, prove Theorem 3.1
C ARGUMENT FOR ACTIVE DATA COLLECTION VIA RANDOMIZED LABELING
In this section, explain in more detail the randomized labeling algorithm described in Section 3.4 We first revisit Thompson sampling. then provide arguments for how our randomized labeling algorithm relates to it, highlight the differences. and then prove regret bound for this scheme under mild assumptions for this algorithm. Our proof follows commonly available proof strategies for Thompson sampling.
15

---

Under review as a conference paper at ICLR 2020
Algorithm 3 Thompson Sampling (TS)
1: Initialize a policy 2'' -> R. data so-far Do ( prior over 0 in fo P(O" (Da)
2: for step in {0, .... T-1) do
3: 0t ~ (FF
(Sample 0, from the posterior)
4: Query Xx arg maxx (x)) 1 8c] (Query based on the posterior probability x; is optimal)
5: Observe outcome: f fxx))
6: D++1 = D, U (xe, fx+))
7: end for
Notation The TS algorithm queries the true function f at locations (Xt)xeN and observes true function values at these points f(x,). The true function f(x) is one of many possible functions that can be defined object, it is common to represent a distribution p' over the true function f. This is justified because, often, multiple parameter assignments f,, can give us the same overall function. We parameterize by a set of parameters l"
the space R!x Instead of representing the true objective function as a point
The T period regret queries X1," xx is given by the random variable
Regret(T) (T) = Σ [f(xt) - f(x))]
1=0
Since selection of X can be stochastic, amalyse Bayes risk Russo & Van Roy 2016, Russo et al., 2018), we define the Bayes risk as the expected regret over randomness in choosing X,, observing
f(x)). ) and over the prior distribution P(0*). This definition is consistent with Russo & Van Roy
(0166).
E(Regret(T)) (T)) = E
Σ [f(x*) - -
Let n=' be the policy with which Thompson sampling queries new datapoints. We do not make any assumptions on the stochasticity of π TS, therefore, it can be stochastic policy in general. However, make 2 assumptions (A1, A2). The same assumptions have been made in Russo & Van Roy (0116).
A1: supx [xx - infx f(x) (Difference between max and min scores is bounded by If this is not true, we can scale the function values so that this becomes true.
A2: Effective size of X is finite.
TS (Alg3) queries the function value at x based on the posterior probability that x is optimal. More formally, the distribution that TS queries x, from can be written as: = = P Px(* (x* = -D.). When we use parameters 0 to represent the function parameter, and thus this reduces to sampling an input that is optimal with respect to the current posterior at each iteration: E arg max [E[fo (x) = 0z).
XEX
MINs (Alg 2 train inverse maps fō (-), parameterized fo (2, where y R. We call an inverse map optimal if it is uniformly optimal given O,, i.e. 11,5, (ınax f(x) ò oargg maxx [xf(())00 S Ex, where e, is controllable (usually the case in supervised learning, errors can be controlled by cross-validation),
Now, we are ready to show that the regret incurred the randomized labelling active data collection scheme is bounded by (VTT). Our proof follows the analysis of Thompson sampling presented in Russo & Van Roy 2016). We first define information ratio and then use it to prove the regret bound.
Information Ratio information gain i.c. the expected reduction in the entropy of the posterior distribution of X
Russo & Van Roy (016)) related the expected regret of TS to its expected
By effective size we refer to the intrinsic dimensionality of X. This doesn't t necessarily imply that X should be discrete. For example, under linear approximation to the score function fo (x). i... if fo(x) = x. this defines polyhedron but just analyzing finite set of just extremal points of the polyhedron works out, thus making [X| effectively finite.
16

---

Under review as a conference paper at ICLR 2020
Information ratio captures this quantity, and is defined as:
B, (xxx) [(x)] Te (x*: (X,, [x,)))
where ((,-) is the mutual information between two random variables and all expectations E, are defined to be conditioned on D,. If the information ratio is small, Thompson sampling can only incur large regret when it is expected to gain lot of information about which x is optimal. Russo & Van Roy (2016) then bounded the expected regret in terms of the maximum amount of information any algorithm could expect acquire, which they observed is at most the entropy of the prior distribution of the optimal x.
Lemma C.1 (Bayes-regret of vanilla TS)so & Van Roy, 2016).. For any T E N, (i.e. information ratio is bounded above) u.s. for each & E .T.....T}.
[Regrer 15) S (X+)T
We refer the readers to the proof of Proposition 1 in Russo & Van Roy (016). The proof presented in Russo & Van Roy (2016) does not rely specifically on the property that the query made by the Thompson sampling algorithm at each iteration x is posterior optimal, but rather it suffices to have a bound on the maximum value of the information ratio r at each iteration 1. Thus, if an algorithm chooses to query the true function at a datapoint X such that these queries always contribute in learning more about the optimal function, i.e. I(---) appearing the denominator of I is always more than threshold, then information ratio is lower bounded, and that active data collection algorithm will have sublinear asymptotic regret. We are interested in the case when the active data collection algorithm queries a datapoint x; at iteration I, such that X, is the optimum for a function fa. where
O, is a sample from the posterior distribution over Or, i.e. Ó, lies in the high confidence region of the posterior distribution over 6g given the data Dj seen so far. In this case, the mutual information between the optimal datapoint x* and the observed (x,, )x:)) input-score pair is likely to be greater than O. More formally,
1xxx", (xt, (xx))) 2 O A xx = arg max Sg, (x) where P(O.(D.) 2 éthreshold
(8)
The randomized labeling scheme for active data collection in MINs performs this step. The algorithm samples bunch of (x,y) datapoints, sythetically generated, for example, in our experiments, we add noise to the values of x, and randomly pair them with unobserved or rarely observed values of y. If the underlying true function f is smooth, then there exist a finite number of points that are sufficient to uniquely describe this function f. One measure to formally characterize this finite number of points that are needed to uniquely identify all functions in function class is given by Eluder dimension (Russo& & Van Roy).
By augmenting synthetic datapoints and training the inverse map on this data, the MIN algorithm ensures that the inverse map is implicitly trained to be an accurate inverse for the unique function fa, that is consistent with the set of points in the dataset D, and the augmented set S.. Which sets of functions can this scheme represent? The functions should be consistent with the data seen so far D., and can take randomly distributed values outside of the seen datapoints. This can roughly argued to be a sample from the posterior over functions, which Thompson sampling would have maintained given identical history D,
Lemma C.2 (Bounded-error training of the posterior-optimal X; preserves asymptotic Bayes-regret). VL E N, let &, be any input such that s(x.) 2 mmax (x)(() Et. MIN chooses to query the true function at &2 and if the sequence (Et)tEN satisfies Σx c = (VT)),
then, the regret from querying this ex-optimal &e which is denoted in general as the policy n 18 is given by TE RegrettT # #T)) = (VVT)
Proof. This lemma intuitively shows that if posterior-optimal inputs x, can be "approximately" queried at each iteration, we can still maintain sublinear regret. To see this, note:
f(x*)) f(x)) f(x*) f(x.) f(x:) f(x)).
17

---

Under review as a conference paper at ICLR 2020
T Regrett T, # AT = Regeett TS]] + Σ -
The second term can be bounded by the absolute value in the worst case, which amounts Ex extra Bayesian regret. As Bayesian regret of TS is (VVT) and E = ( the new overall regret is also (VTT).
Theorem C.3 (Bayesian Regret of randomized labeling active data collection scheme proposed in Section 3.4 is O(VT)), Regret incurred by the MIN algorithu with randomized labeling is of the order O (TH(())) + C)T).
Proof. Simply put, we will combine the insight about the mutual information I(x", (Xe, (xx))) > 0 and C.2 in this proof. Non-zero mutual information indicates that achieve ((VT) regret if we query x,S which are optimal corresponding to some implicitly defined forward function lying in the high confidence set of the true posterior given the observed datapoints D,. Lemma C.2 says that if bounded errors made in fitting the inverse map, the overall regret remains (VVT).
More formally, if 115 (maxx f(x)(0,) 8 ('rrg maxx (x) 0. - 8z, this means that E M (f(x)) - = [f())ll & ll 11,o,
TS| S 6e Rmax S Ex
and now application of Lemma C.2 gives us the extra regret incurred. (Note that this also provides us way to choose the number of training steps for the inverse map)
Further, note if we sample x, at iteration from a distribution that shares support with the true posterior over optimal xg (which used by TS), we still incur sublinear, bounded O (A+)) regret.
In the worst case, the overall bias caused due to the approximations will lead to an additive cu- mulative increase in the Bayesian regret, and hence, there is constant C > such that E[Regret ((V(TH((*) CYT).
D ADDITIONAL EXPERIMENTS AND DETAILS
D.I CONTEXTUAL IMAGE OPTIMIZATION
2
2
8
3
O
In this set of static dataset experiments, we study contextual MBO tasks on image pixels. Un- like the contextual bandits case, where x corre- sponds to an image label, here x corresponds to entire images. We construct several tasks. First, we study stroke width optimization on MNIST characters, where the context is the class of the digit we wish to optimize. Results are shown in Figure MINs correctly produce digits of the right class, and achieve an average score over the digit classes of 237.6, whereas the average score of the digits in the dataset is 149.0.
4
L1
4
5
3
3
0
^
O
2
7
T
7 3
4 /
M
7
3
2
(a)
Groundtruth
S
Context Input
9
6
3
-Produced x
6
6
6
2
R
2
2
8
3
5
The next task to test the ability of MINs to be able to complete/inpaint unobserved patches of an image given an observed context patch. We use two masks: mask A: only top half and mask B: only top one-fourth parts of the image are visible, mask out portions of the image and present the masked image as context e to the MIN, with the goal being produce valid completion x, while still maximizing score cor- responding the stroke width. We present some
9
5
(c)
(6)
Figure Contextual MBO on MNIST. In (a) and (b), top half and top one-fourth of image respectively and in (c) the one-hot encoded label are provided as contexts. The goal is to produce the maximum stroke width character that is valid given the context. In (a) and (b), show triplets of the groundtruth digit (green), the context passed as input (yellow) and the produced images x from the MIN model (purple).
18

---

Under review as a conference paper at ICLR 2020
Figure 5: Additional results for non-contextual image optimization. This task is performed on the CelebA dataset. The aim is to maximize the score of image which is given by the sum of altributes: eyeglasses, smiling, wavy hair and no beard. MINs produce optimal visually these solutions indeed optimize the score.
sample completions in Figure | The quantitative results are presented Table 6
We find that MINs effective as compared completions for the context in the dataset in terms of score while still producing a visibly valid character.
We evaluate MINs
complex semantic optimization task the CelebA (iu al. 2015) dataset. We choose a subset of attributes and provide their one-hot encoding as context to the model. The score is equal to the & norm of the binary indicator vector for different subset of attributes disjoint from the context. We present our results in Figure 3 We observe that MINs produce diverse images consistent with the context, and is also able to effectively infer the score function, and learn features to maximize it. Some of the model produced optimized solutions were presented in Section 4 in Figure S In this section, present the produced generations for some other contexts. Figure shows these results.
D.2 ADDITIONAL RESULTS FOR NON-CONTEXTUAL. IMAGE OPTIMIZATION
In this section, we present some additional results for non-contextual image optimization problems. We also evaluated our contextual optimization procedure on the CelebA dataset in non-contextual setting. The reward function is the same as that in the contextual setting - the sum of attributes: wavy hair, beard, smiling and eyeglasses. We find that MINs are able to sucessfully produce solutions in this scenario as well. We show some optimized outputs at different iterations from the model in Figure
cGAN baseline. We compare our MIN model to a cGAN baseline on the IMDB-Wiki faces dataset for the semantic age optimization task. general, we found that the cGAN model learned to ignore the score value passed as input even when trained on the entire dataset (without excluding the youngest faces) and behaved almost like a regular unconditional GAN model when queried to produce images corresponding the fact that age of a person doesn't have enough direct signal to guide the model to utilize unless other tricks like reweighting proposed in Section 3.33 which explicitly enforce the model attention to datapoints of interest, are used. We present the produced optimized x in Figure
the smallest age. We suspect that this could possibly be due to
D.3 QUANTITATIVE SCORES FOR NON-CONTEXTUAL MNIST OPTIMIZATION
In Figure S we highlight the quantitative score values for the stroke width score function (defined the number of pixels which have intensity more than threshold). Note that MINs achieve the highest value of average score while still resembling a valid digit, that stays inside the manifold of valid digits, unlike forward model which can get high values of the score function (number of pixels turned on), but doesn"t 't stay on the manifold of valid digits.
D.4 EXPERIMENTAL DETAILS AND SETUP
19

---

Under review as a conference paper at ICLR 2020
Figure 6: Optimal x solutions produced by cGAN youngest face optimization task on IMDB-face dataset. We note that a cGAN learned to ignore the score value and produced images as an unconditional model, without any noticeable correlation with the score value. The samples produced mostly correspond to the most froquently occurring images in the dataset.
Bang
Brown or Black Hair
Both Brown/Black Hair
Moustache
Figure 7: Images returned by the MIN optimization for optimization over images. We note that MINs perform successful optimization over the an objective defined by the sum of desired attributes. Moreover, for unseen contexts, such as both brown and black hair, the optimized solutions look aligning with the context reasonahly. and optimize for the score well.
MIN MIN-I MIN-R F
MIN
MIN-I MIN-R
F
MIN MIN-T
MIN-R
$
3
2
3
3
3
1
4
7
3
3
3
3
3
2
3
8
7
8
7
8
8
2
3
3
3
8
7
6
9
276.3
341.3
242.40 453.2
255.4
240.8
333,4
515.7
2.32
1.12
1.53
1.42
(a) Thickest stroke
(b) Thickest digit (3)
(c) Most number of blobs (8)
Figure 8: Results for non-contextual static dataset optimization on MNIST annotated with quantitative score values achieved mentioned below each figure.
Table 6: Average quantitative perfor-
In this section, we explain the experimental details and the setup of model. For our experiments involving MNIST and optimiza- tion of benchmark functions task, we used the same architecture fully connected GAN where the generator and discriminator are both fully connected networks. We based our code for this part on the open-source implementation (Linder-Norén). For the mask B
mance for MNIST inpainting
Mask mask A
MIN
Dataset
223.57
149.0
234.32
149.0
forward model experiments in these settings, we used a 3-layer feedforward ReL.U network with hidden units of size 256 each in this setting. For all experiments on CelebA and IMDB-WiL Wiki faces, we used the VGAN (Peng et al. (201)) model and the associated codebase as our starting setup. For experiments on batch contextual bandits, we used a fully con- nected discriminator and generator for MNIST, and convolutional generator and Resnet 8-like discriminator for CIFAR-10. 10. The prediction in this setting is categorical - of 10 labels needs to be predicted, instead of using reinforce or derivative free optimization train the inverse map. we used the Gumbel-soffmax, Jang et al. (2016) trick with a temperature т = 0.75, to be able to use stochastic gradient descent train the model. For the protein flourescence maximization experiment,
20

---

Under review as a conference paper at ICLR 2020
used 2-layer, 256-unit feed-forward gumbel-softmat inverse map and 2-layer feed- forward discriminator.
We trained models present in open-souree implementations of BanditNet (Sachdeva), but were unable to reproduce results as reported by Joachims et al. (2018). Thus we reported the paper reported numbers from the BanditNet paper in the main text well.
Temperature hyperparameter 7 which is used to compute the reweighting distribution is adaptively chosen based the 90th percentile score in the dataset. For example, if the difference between Ymaz and perceneeee is given by a, we choose r = a. This scheme can adaptively change temperatures in the active setting. In order to select the constant which decides whether the bin corresponding to Ny use densities rather than particular value of y is small or not, we first convert the expression Ny+A
( where ýD (y) is the empirical density of observing in D, and now we use the same constant À = 0.003. We did not observe a lot of sensitivity to À values in the range [0.0001, 0.007], all of which performed reasonably similar. We usually fixed the number bins to 20 for the purposed of reweighting, however note that the inverse map was still trained on continuous y values, which helps it extrapolate.
absolute counts, that is, An(y)
In the active setting, train two copies of f side by side. One of them is trained on the augmented datapoints generated out of the randomized labelling procedure, and the other copy is just trained the real datapoints. This done so to prevent instabilities while training inverse maps. Training can also be made more incremental in this manner, and we need to train an inverse map to optimality inside every iteration of the active MIN algorithm, but rather we can train both the inverse maps for a fixed number of gradient steps.
21


Reviews:
[{"review_id": "SyeB72uRKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper prposes to learn an inverse network to predict x given a target y for optimisation, instead of the traditional way of optimisation (e.g. using Bayesian optimisation for the complex cases considered in the paper). However, unfortunately, this paper is too close in concept, and in my understanding lower in the solution quality to this recent paper:\n\nNguyen, Phuoc, Truyen Tran, Sunil Gupta, Santu Rana, Matthew Barnett, and Svetha Venkatesh. \"Incomplete conditional density estimation for fast materials discovery.\" In Proceedings of the 2019 SIAM International Conference on Data Mining, pp. 549-557. Society for Industrial and Applied Mathematics, 2019.\n\nPlease let me know if I missed anything. Otherwise it is a reject from me.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyxTb_1TYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors present a method for model based blackbox optimization for cases where the design variables x are likely to lie on a manifold within a high dimensional space. The basic idea is to use a generative model to map targets y to potential design variables that would produce that target, and use this generative model in combination with a trained forward model to produce an optimal setting x* that both lies on the manifold and takes a high y* value, possibly higher than seen in the collected dataset (see the first two terms in equation 4).\n\nOverall, I feel positively about the paper, largely because of the idea it introduces. In particular, the task of manifold constrained optimization is not highly studied in BayesOpt, and it probably should be. Specifically, BayesOpt *is* occasionally studied on problems where the inputs are natural images, protein sequences, controller policies, and other settings where not all inputs are going to be valid, even if all inputs can technically give rise to a function value. The use of a GAN to constrain the possible inputs via a learned measure p(x|y) is a nice idea where applicable, and represents a concrete utility for GANs beyond generating images. All the individual components the authors' use to put this together are somewhat basic; however, the experimental results are vaguely convincing enough and I would choose to evaluate the merits of this paper on the core idea presented rather than whether or not the latest GAN architecture was used in the actual implementation.\n\nWith that said, I have a few comments. First, the constructions in 3.2 and 3.3 are perhaps a little more ad hoc than they need to be. If we are free to use probabilistic machinery for the forward model, then the fact that the inverse model effectively gives us a distribution fit to p(x|y) suggests that this can be naturally incorporated in to existing BayesOpt schemes. It might seem almost more satisfying to see a simple existing bayesopt pipeline extended with this idea, rather than wholly inference/recommendation and acquisition schemes introduced.\n\nAdditionally, two of the sections in the experimental results seemed somewhat rushed and need significant additional detail. The setup in the first portion of 4.1 is largely left to the reader to read Joachims et al., 2018 -- I'd like to see a more self contained description of the task. The more egregious example of this is the \"protein floresence [sic] maximization\" section. One of the baseline methods (GB) exists only as an acronym in Table 3 with no citation or discussion, and in general the section could be significantly expanded.\n\nThis last paragraph is something of a shame and one of the main weaknesses I see with the paper, as these two section are among the only quantitative results for the authors' optimization algorithm. Results are reported on Branin and Hartmann6, but frankly I don't see the value of these because they are far outside the intended application domain for the authors' work, as they are functions that are specifically intended to be optimized over compact domains. In general, it would be better to see more emphasis placed on quantitative experiments, particularly when the task is optimization. Perhaps adversarial image generation (where x arguably leaves the manifold of natural images, but not so far as to be random noise) or some other image task might substitute for the benchmark functions?\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xX0KDRFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of solving a black-box optimization problem where only some samples have been observed. This task requires a good model that can be both expressive and generalizable. Instead of learning only a single forward model of x -> y, this paper proposes to additionally use a mapping from y -> x. Optimizing in the space of z instead of x can be much simpler, and this should also act as a strong regularizer during training. Specifically, the paper uses a GAN that transforms [y,z] -> x, where z is stochastically sampled. This paper further proposes a reweighting scheme that interpolates between a uniform weighting and weighting the best sample so far, as well as a sampling procedure that iteratively samples points and refits a second model, which was inspired by Thompson sampling.\n\nPros:\n - The proposed idea of using an inverse mapping is straightward but shown to be effective. The methods to make this work, namely reweighting and the randomized labeling procedure, seem to have some amount of theory behind them, though their presentation was confusing without multiple read-throughs. \n - There are plenty of experiments across a wide array of domains, including images, 2D and 6D functions, and proteins which have a discrete representation.\n - There are some comparisons to Spearmint and a scalable BO method (DNGO), though only on the 2D and 6D functions.\n\nCons:\n - The proposed pieces were often difficult to follow, and there doesn't seem to be sufficient information regarding the reweighting and randomized labeling for understanding and reproducing this work (see Questions). \n - Only a few ablation experiments were carried out, and the effect of reweighting seems to only appear as a visual comparison in Figure 1. As the method seems quite different from related approaches, having more systematic comparisons would give us a better understanding of the core sources of improvement and better instigate future works.\n\nQuestions:\n- After Theorem 3.1, what is g? A linear function can only change a pdf by its bias (as any multiplication only affects the normalization constant), so not sure how to interpret g. I wasn't able to find this in the Appendix either.\n- When creating the augmented dataset for defining & approximating some kind of posterior, can you be more concrete about how the y (and x) values are chosen? More specifically, is this procedure approximating some kind of meaningful posterior? If the y values are sampled randomly (from some fixed prior or perhaps from the p(y) described in the reweighting section?), what makes this procedure meaningful (does it rely on the learned GAN somehow)? Seeing a simple example in 1D would be a good visualization. \n- What are the different rows of Figure 1 (different initializations)? Why are the different rows so similar for MIN but not for F? Also, what were the original examples (are they closer to the F results, or the MIN results)?\n- For the protein task, how did you structure the output of f^-1(x) and how did you backpropagate through the discrete variables?\n- In the experiments, it wasn't explicitly clear what the \"MIN without inference\" setting referred to; I assume this is where a single sample from the inverse mapping GAN was used?\n\nAdditional Comments:\n- When discussing the reweighted objective, the notation is a little confusing. The variables (j,k) is a different parameterization of the index i, but initially I thought k was indexing features of x. Perhaps a brief explanation of the rearrangement would make this clearer at first glance, or even remove this from the main text? (Without going through the proof, it isn't clear why this rearrangement is discussed.)\n- After (1), it wasn't yet clear why this is called \"model-based\" optimization, as no model has been introduced yet.\n- Are BO methods truly not applicable to the static setting? The static setting seems to be just a single prediction, which seems like a single step (ie. special case) of the non-static case?\n- typo: \"method to solve perform optimization\"\n- typo: Figure 1 \"Obsere\"", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes Model Inversion Networks (MINs) to solve model optimization problems high-dimensional spaces. The paper received three reviews from experts working in this area. In a short review, R1 recommends Reject based on limited novelty compared to an ICDM 2019 paper. R2 recommends Weak Reject, identifying several strengths of the paper but also a number of concerns including unclear or missing technical explanations and need for some additional experiments (ablation studies). R3 recommends Weak Accept, giving the opinion that the idea the paper proposes is worthy of publication, but also identifying a number of weaknesses including a "rushed" experimental section that is missing details, need for additional quantitative experimental results, and some "ad hoc" parts of the formulation. The authors prepared responses that address many of these concerns, including a convincing argument that there is significant difference and novelty compared to the ICDM 2019. However, even if excluding R1's review, the reviews of R2 and R3 are borderline; the ACs read the paper and while they feel the work has significant merit, they agree with R2 and R3 that the paper needs additional work and another round of peer review to fully address R2 and R3's concerns. 


Decision: Reject