Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper prposes to learn an inverse network to predict x given a target y for optimisation, instead of the traditional way of optimisation (e.g. using Bayesian optimisation for the complex cases considered in the paper). However, unfortunately, this paper is too close in concept, and in my understanding lower in the solution quality to this recent paper:

Nguyen, Phuoc, Truyen Tran, Sunil Gupta, Santu Rana, Matthew Barnett, and Svetha Venkatesh. "Incomplete conditional density estimation for fast materials discovery." In Proceedings of the 2019 SIAM International Conference on Data Mining, pp. 549-557. Society for Industrial and Applied Mathematics, 2019.

Please let me know if I missed anything. Otherwise it is a reject from me.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper tackles the problem of solving a black-box optimization problem where only some samples have been observed. This task requires a good model that can be both expressive and generalizable. Instead of learning only a single forward model of x -> y, this paper proposes to additionally use a mapping from y -> x. Optimizing in the space of z instead of x can be much simpler, and this should also act as a strong regularizer during training. Specifically, the paper uses a GAN that transforms [y,z] -> x, where z is stochastically sampled. This paper further proposes a reweighting scheme that interpolates between a uniform weighting and weighting the best sample so far, as well as a sampling procedure that iteratively samples points and refits a second model, which was inspired by Thompson sampling.

Pros:
 - The proposed idea of using an inverse mapping is straightward but shown to be effective. The methods to make this work, namely reweighting and the randomized labeling procedure, seem to have some amount of theory behind them, though their presentation was confusing without multiple read-throughs. 
 - There are plenty of experiments across a wide array of domains, including images, 2D and 6D functions, and proteins which have a discrete representation.
 - There are some comparisons to Spearmint and a scalable BO method (DNGO), though only on the 2D and 6D functions.

Cons:
 - The proposed pieces were often difficult to follow, and there doesn't seem to be sufficient information regarding the reweighting and randomized labeling for understanding and reproducing this work (see Questions). 
 - Only a few ablation experiments were carried out, and the effect of reweighting seems to only appear as a visual comparison in Figure 1. As the method seems quite different from related approaches, having more systematic comparisons would give us a better understanding of the core sources of improvement and better instigate future works.

Questions:
- After Theorem 3.1, what is g? A linear function can only change a pdf by its bias (as any multiplication only affects the normalization constant), so not sure how to interpret g. I wasn't able to find this in the Appendix either.
- When creating the augmented dataset for defining & approximating some kind of posterior, can you be more concrete about how the y (and x) values are chosen? More specifically, is this procedure approximating some kind of meaningful posterior? If the y values are sampled randomly (from some fixed prior or perhaps from the p(y) described in the reweighting section?), what makes this procedure meaningful (does it rely on the learned GAN somehow)? Seeing a simple example in 1D would be a good visualization. 
- What are the different rows of Figure 1 (different initializations)? Why are the different rows so similar for MIN but not for F? Also, what were the original examples (are they closer to the F results, or the MIN results)?
- For the protein task, how did you structure the output of f^-1(x) and how did you backpropagate through the discrete variables?
- In the experiments, it wasn't explicitly clear what the "MIN without inference" setting referred to; I assume this is where a single sample from the inverse mapping GAN was used?

Additional Comments:
- When discussing the reweighted objective, the notation is a little confusing. The variables (j,k) is a different parameterization of the index i, but initially I thought k was indexing features of x. Perhaps a brief explanation of the rearrangement would make this clearer at first glance, or even remove this from the main text? (Without going through the proof, it isn't clear why this rearrangement is discussed.)
- After (1), it wasn't yet clear why this is called "model-based" optimization, as no model has been introduced yet.
- Are BO methods truly not applicable to the static setting? The static setting seems to be just a single prediction, which seems like a single step (ie. special case) of the non-static case?
- typo: "method to solve perform optimization"
- typo: Figure 1 "Obsere"
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors present a method for model based blackbox optimization for cases where the design variables x are likely to lie on a manifold within a high dimensional space. The basic idea is to use a generative model to map targets y to potential design variables that would produce that target, and use this generative model in combination with a trained forward model to produce an optimal setting x* that both lies on the manifold and takes a high y* value, possibly higher than seen in the collected dataset (see the first two terms in equation 4).

Overall, I feel positively about the paper, largely because of the idea it introduces. In particular, the task of manifold constrained optimization is not highly studied in BayesOpt, and it probably should be. Specifically, BayesOpt *is* occasionally studied on problems where the inputs are natural images, protein sequences, controller policies, and other settings where not all inputs are going to be valid, even if all inputs can technically give rise to a function value. The use of a GAN to constrain the possible inputs via a learned measure p(x|y) is a nice idea where applicable, and represents a concrete utility for GANs beyond generating images. All the individual components the authors' use to put this together are somewhat basic; however, the experimental results are vaguely convincing enough and I would choose to evaluate the merits of this paper on the core idea presented rather than whether or not the latest GAN architecture was used in the actual implementation.

With that said, I have a few comments. First, the constructions in 3.2 and 3.3 are perhaps a little more ad hoc than they need to be. If we are free to use probabilistic machinery for the forward model, then the fact that the inverse model effectively gives us a distribution fit to p(x|y) suggests that this can be naturally incorporated in to existing BayesOpt schemes. It might seem almost more satisfying to see a simple existing bayesopt pipeline extended with this idea, rather than wholly inference/recommendation and acquisition schemes introduced.

Additionally, two of the sections in the experimental results seemed somewhat rushed and need significant additional detail. The setup in the first portion of 4.1 is largely left to the reader to read Joachims et al., 2018 -- I'd like to see a more self contained description of the task. The more egregious example of this is the "protein floresence [sic] maximization" section. One of the baseline methods (GB) exists only as an acronym in Table 3 with no citation or discussion, and in general the section could be significantly expanded.

This last paragraph is something of a shame and one of the main weaknesses I see with the paper, as these two section are among the only quantitative results for the authors' optimization algorithm. Results are reported on Branin and Hartmann6, but frankly I don't see the value of these because they are far outside the intended application domain for the authors' work, as they are functions that are specifically intended to be optimized over compact domains. In general, it would be better to see more emphasis placed on quantitative experiments, particularly when the task is optimization. Perhaps adversarial image generation (where x arguably leaves the manifold of natural images, but not so far as to be random noise) or some other image task might substitute for the benchmark functions?


review_title: Official Blind Review #3

Meta-review: This paper proposes Model Inversion Networks (MINs) to solve model optimization problems high-dimensional spaces. The paper received three reviews from experts working in this area. In a short review, R1 recommends Reject based on limited novelty compared to an ICDM 2019 paper. R2 recommends Weak Reject, identifying several strengths of the paper but also a number of concerns including unclear or missing technical explanations and need for some additional experiments (ablation studies). R3 recommends Weak Accept, giving the opinion that the idea the paper proposes is worthy of publication, but also identifying a number of weaknesses including a "rushed" experimental section that is missing details, need for additional quantitative experimental results, and some "ad hoc" parts of the formulation. The authors prepared responses that address many of these concerns, including a convincing argument that there is significant difference and novelty compared to the ICDM 2019. However, even if excluding R1's review, the reviews of R2 and R3 are borderline; the ACs read the paper and while they feel the work has significant merit, they agree with R2 and R3 that the paper needs additional work and another round of peer review to fully address R2 and R3's concerns. 


Decision: Reject