Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, the authors consider non-sequential (in the sense that many hyperparameter evaluations are done simultaneously) and uninformed (in the sense that the hyperparameter evaluations are chosen independent of the validation errors observed) hyperparameter search using determinantal point processes (DPPs). DPPs are probability distributions over subsets of a ground set with the property that subsets with more "diverse" elements have higher probability. Diverse here is defined using some similarity metric, often a kernel. Under the RBF kernel, the more diverse a set of vectors is, the closer the kernel matrix becomes to the identity matrix, and thus the larger the determinant (and therefore probability under the DPP) grows. The authors propose to do hyperparameter tuning by sampling a set of hyperparameter evaluations from a DPP with the RBF kernel.

Overall, I have a couple of concerns about novelty as well as the experimental evaluation for the authors to address. As the authors rightly point out, sampling hyperparameter values from a DPP is equivalent to sampling proportional to the posterior uncertainy of a Gaussian process, effectively leading to a pure exploration algorithm. As the authors additionally point out, such methods have been considered before, including methods that directly propose to batch Bayesian optimization by choosing a single exploitative point and sampling the remainder of the batch from a DPP (e.g., [Kathuria et al., 2016]). The default procedure for parallel BayesOpt used by SMAC [R2] is (I believe) also to choose a purely explorative batch. I am unconvinced by the argument that "while this can lead to easy parallelization within one iteration of Bayesian optimization, the overall algorithms are still sequential." These methods can typically be expanded to arbitrarily large batches and fully utilize all parallel hardware. Most implementations of batch Bayesian optimization in practice (SMAC and Spearmint as examples) will even start new jobs immediately as jobs finish -- these implementations do not wait for the entire batch to finish typically.

Additionally, while there has been some work extending GP-based BayesOpt to tree-based parameters [R3], at a minimum SMAC in particular is known well suited to the tree-based parameter search considered by the authors. I am not sure that I agree that TPE is state-of-the-art on these problems: SMAC typically does much better in my experience. 

Ultimately, my concern is that--considering these tools are open source and relatively stable software at this point--if DPP-only based hyperparameter optimization is truly better than the parallelization approach of SMAC, it should be straightforward enough to download SMAC and demonstrate this. If the argument that BayesOpt is somehow "still sequential" is true, then k-DPP-RBF should outperform these tools in terms of wall clock time to perform optimization, correct?

[R1] Kathuria, Tarun and Deshpande, Amit and Kohli, Pushmeet. Batched Gaussian Process Bandit Optimization via Determinantal Point Processes, 2016.

[R2] Several papers, see: http://www.cs.ubc.ca/labs/beta/Projects/SMAC/

[R3] Jenatton, R., Archambeau, C., González, J. and Seeger, M., 2017, July. Bayesian Optimization with Tree-structured Dependencies. In International Conference on Machine Learning (pp. 1655-1664).
review_title: An interesting idea for pure exploration hyperparameter tuning, but needs to be compared to more recent existing methods.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose k-DPP as an open loop (oblivious to the evaluation of configurations) method for hyperparameter optimization and provide its empirical study and comparison with other methods such as grid search, uniform random search, low-discrepancy Sobol sequences, BO-TPE (Bayesian optimization using tree-structured Parzen estimator) by Bergstra et al. (2011). The k-DPP sampling algorithm and the concept of k-DPP-RBF over hyperparameters are not new, so the main contribution here is the empirical study. 

The first experiment by the authors shows that k-DPP-RBF gives better star discrepancy than uniform random search while being comparable to low-discrepancy Sobol sequences in other metrics such as distance from the center or an arbitrary corner (Fig. 1).

The second experiment shows surprisingly that for the hard learning rate range, k-DPP-RBF performs better than uniform random search, and moreover, both of these outperform BO-TPE (Fig. 2, column 1).

The third experiment shows that on good or stable ranges, k-DPP-RBF and its discrete analog slightly outperform uniform random search and its discrete analog, respectively.

I have a few reservations. First, I do not find these outcomes very surprising or informative, except for the second experiment (Fig. 2, column 1). Second, their study only applies to a small number like 3-6 hyperparameters with a small k=20. The real challenge lies in scaling up to many hyperparameters or even k-DPP sampling for larger k. Third, the authors do not compare against some relevant, recent work, e.g., Springenberg et al. (http://aad.informatik.uni-freiburg.de/papers/16-NIPS-BOHamiANN.pdf) and Snoek et al. (https://arxiv.org/pdf/1502.05700.pdf) that is essential for this kind of empirical study. 

  
review_title: comparison with recent work and scalability

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
This paper considers hyperparameter searches in which all of the
candidate points are selected in advance.  The most common approaches
are uniform random search and grid search, but more recently
low-discrepancy sequences have sometimes been used to try to achieve
better coverage of the space.  This paper proposes using a variant of
the determinantal point process, the k-DPP to select these points.
The idea is that the DPP provides an alternative form of diversity to
low-discrepancy sequences.

Some issues I have with this paper:

1. Why a DPP? It's pretty heavyweight. Why not use any of the other
(potentially cheaper) repulsive point processes that also achieve
diversity?  Is there anything special about it that justifies this
work?

2. What about all of the literature on space-filling designs, e.g.,
latin hypercube designs?  Statisticians have thought about this for a
long time.

3. The motivation for not using low-discrepancy sequences was discrete
hyperparameters.  In practice, people just chop up the space or round.
Is a simple kernel with one length scale on a one-hot coding adding
value? In this setup, each parameter can only contribute "same or
different" to the diversity assessment.  In any case, the evaluations
didn't have any discrete parameters.  Given that the discrete setting
was the motivation for the DPP over LDS, it seems strange not to even
look at that case.

4. How do you propose handling ordinal variables? They're a common
case of discrete variables but it wouldn't be sensible to use a
one-hot coding.

5. Why no low discrepancy sequence in the experimental evaluation of
section 5?  Since there's no discrete parameters, I don't see what the
limitation is.

6. Why not evaluate any other low discrepancy sequences than Sobol?

7. I didn't understand the novelty of the MCMC method relative to
vanilla M-H updates.  It seems out of place.

8. The figures really need error bars --- Figure 3 in particular.  Are
these differences statistically significant?

review_title: Limited Scope, Weak Evaluation

Meta-review: The idea of using the determinant of the covariance matrix over inputs to select experiments to run is a foundational concept of experimental design.  Thus it is natural to think about extending such a strategy to sequential model based optimization for the hyperparameters of machine learning models, using recent advances in determinantal point processes.  The idea of sampling from k-DPPs to do parallel hyperparameter search, balancing quality and diversity of expected outcomes, seems neat.  While the reviewers found the idea interesting, they saw weaknesses in the approach and most importantly were not convinced by the empirical results.  All reviewers thought that the baselines were inappropriate given recent work in hyperparameter optimization (and classic work in statistics).

Pros:
- Useful to a large portion of the community (if it works)
- An interesting idea that seems timely

Cons:
- Only slightly outperforms baselines that are too weak
- Not empirically compared to recent literature
- Some of the design and methodology require more justification
- Experiments are limited to small scale problems
Decision: Reject