Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper presents novel Langevin sampling implementations with (i) zero order and (ii) deep neural network gradient approximations for constraint satisfaction problems. These approximations improve the computational efficiency by a factor of 17  compared to related black box sampling approaches (Shen et al.). 
As a non-domain expert I cannot evaluate the importance of the contributions or the complexity of the tasks. However, given the well written and clearly structured presentation, the theoretical proofs and the generality of the constraint solver (with learned approximated gradients), I would vote for accepting the paper. 

Open questions:
- How does the approach relate to "multi-objective Bayesian optimization". The result of this multi objective optimization problem can be also used to further investigate different configurations in different tasks  (generalization via pareto front sampling)?

- The authors note that their approach is closely related to Shen et al. (2019). However, there is no algorithmic comparison in the experiments. Can the work of Shen et al. (2019) also be applied to the constraint setting? 

- How crucial or restricting is the assumption of log-concave and smooth target distributions? Which constraint satisfaction problems fall into this class? Can you provide some examples?  

- The authors use for approximating the gradients a neural network with 4 hidden layers (with 128, 72, 64 and 32 ReLU neurons) and fine tune the networks in a grid search manny. Which parameters were fine tuned? What were the input and output dimensions. I assume 10 kappa and 10 sigma values were the predicted outputs in individual networks. 

Minor Issues:
- Some paragraph titles to no end with a dot, e.g., "Optimization approaches" in page 8
- The last sentence in page 8 is incomplete. 
- The computational speed up is a factor of 17 (page 9) or 15 (page 10)?
- Typo in page 13: after the target output is properly normalization
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper considers the problem of sampling points from a constrained set in R^d where the constraints can only be accessed in a zero order fashion. They consider the specific situation where the constraints are a solution of a complicated PDE solver and hence the derivatives or specific functional forms of the constraint cannot be obtained. They repose the problem as sampling from a Gibbs distribution whose potential contains constraints as penalties in a Lagrangian fashion. They now wish to sample from the Gibbs distribution using Langevin diffusion.  The Langevin process requires a derivative of the gradient. The setting does not allow for that and therefore the authors propose two approaches - 

1. Constructing the gradient from zeroth order entries of a gaussian smoothed potential (much like works of Nesterov et al on zero order optimization). 
2. Using a parameteric function class (like an RKHS or a neural network) to learn a function which well approximates the gradient of the constraints as well given zeroth order constraint evaluations. 

For the latter, the authors propose two approaches. Hermite learning which directly approximates the error on the gradient as well as the function evaluation. However this involves a separate estimate of the gradient (via a zero order approach such as Gaussian smoothing). An alternative which seems more sample efficient is to do direct learning from zero order samples by penalizing first-order Taylor approximation between given points. 

The authors provide a theoretical analysis of the all the approaches and sub-approaches given above and further provide experiments to evaluate this on a problem from material design. I am not at all familiar with the domain of the experiments to comment on competing approaches. As far as I can see the authors also compare only their own proposed algorithms which are many. I will focus on the theoretical analysis which seems to form the bulk of the paper anyway. 

The theoretical analysis seems quite rigorous as it begins by first providing a basic guarantee for constrained langevin sampling when gradients are computed with error. The non error gradient part of this analysis has been established before and the authors mention the references appropriately. I have a couple of questions regarding the precise statements of the theorem that i will ask towards the end of the review. Overall its hard to comment on the tightness of the analysis as the non-error versions are also unclear of the tightness of the bounds. Nevertheless the bounds achieved do not look much worse than the non-error counterparts and are easy to implement. The rest of the bounds focus on achieving low error in approximation of the gradients in various settings. Overall the theory in this part seems very loose in terms of bounds as exponential factors in dimensions start to appear and in that regard seems quite preliminary but its hard to comment on whether its natural or can be improved. 

Overall the paper is a rigorous treatment of multiple components that would go into the problem of sampling zero order constrained sets and merges many ideas which can all be useful. Nevertheless the paper is a little lacking of novelty in the sense that it brings together many existing ideas and provides an analysis of the effect of bringing them together but none of the theory significantly improves over the existing theory.

Some questions I have regarding the theorem statements 

Regarding theorem 1 SPLMC convergence  (and corollaries of the theorem) -  I find it surprising that there is no lower bound assumption on eta in terms of K - only an upper bound. This seems wrong particularly as the theorem as stated then allows eta to be set extremely small while K is finite, in which case there should be no convergence theorems at all. The condition on eta should be a theta(f (K)) for some f type of condition like in the second part of the theorem. I would suggest the authors to look into the theorem - or claify why this is the case. 

I am confused by the presentation of the Shi et al results as there is no penalty appearing for the approximation error due to an RKHS, only a finite sample penalty. Does the result assume that phi belongs to the function class of the RKHS in question? Probably yes and in that case that should be specified. 

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes a solution to overcome the challenges due to the black-box nature of physical constraints that are involved in the design of nano-porous templates with optimal thermoelectric efficiency. 

Unfortunately, I cannot comment on the overall scientific contribution of the paper, as I do not possess the expertise to judge it accurately. My expertise is so outside of this field that I will rely on the judgement of the other reviewers, whom I hope will have more experience and will better know the literature. 

I can only report that the proposed method does not seem to be a particularly good approximation to the zeroth-order method since the mean values for kappa and sigma in table 2 are quite a bit worse than those obtained with the baseline. Of course, the proposed approach is quite a bit faster. However, the paper does not provide a sense of whether these values are actually useful. In practice, would one want to wait longer to get a better quality result, or are the numbers obtained with the proposed approach usable? 

Also, could the proposed approach be applied to other problems? It would be great to see at least one or two other areas where this could be applied to, since I doubt the general ICLR audience is well-versed in nano-porous templates. 
review_title: Official Blind Review #2

Meta-review: The paper is not overly well written and motivated. A guiding thread through the paper is often missing. Comparisons with constrained BO methods would have improved the paper as well as a more explicit link to multi-objective BO. It could have been interesting to evaluate the sensitivity w.r.t. the number of samples in the Monte Carlo estimate. What happens if the observations of the function are noisy? Is there a natural way to deal with this?
Given that the paper is 10+ pages long, we expect a higher quality than an 8-pages paper (reviewing and submission guidelines). 
Decision: Reject