Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper presents an analogous optimization algorithm that is inspired by MCMC. The algorithm maintains a series of weight replicas and sweeps them according to loss functions, based on rules from statistical physics.

I appreciate the idea of introducing temperature here, and the part that describes the idea is nicely written. I may have concerns, as in training neural network, most optimization algorithms use stochastic gradient which intrinsically contain noise inside, although certain level of gradient noise helps to escape saddle point on the landscape, there is no necessary analysis whether this additional noise would contribute to the optimization positively or negatively, after all the noise level has to be bounded for the optimizer to converge.

The experiments are a little insufficient, as the authors used very small datasets, and there is no comparison with other carefully tuned optimization algorithms, which, circle back to my previous concern, is that whether such an algorithm could actually outperform a carefully tuned SGD, ADAM.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: REVIEW 2

The paper proposes a new paradigm to perform hyperparameter search by proposing a way to jointly optimize over the hyperparameter space and the parameter space as opposed to the traditional way to performing these in separation (with the hyperparameter search invoking the parameter optimization) The paper’s main claim is that this allows the seach to follow non-local paths in the joint space. The main methodology proposed in the paper is inspired by the idea of parallel tempering from physics. The paper proposes to view parameter learning under a certain hyperparameter as running a langevin chain at a particular temperature. This is motivated by considering common hyperparameters as batch size, dropout rate or learning rate as inducing a specific level of noise to the training process, the variance of which is analogous to the inverse temperature in Langevin diffusion. 

Under the above analogy, the paper proposes to run an analogue of parallel tempering in the following way. Consider running parallel threads of parameter optimization for various fixed choices of hyperparameter values (each representing a temperature). After running certain number of steps of SGD in each configuration, the paper proposes to exchange hyperparameters within a pair of threads, with the chance of exchange depending upon a criteria motivated by metropolis hastings.  

While the paper proposes an interesting idea I have many reservations about the paper and applicability of the idea in practice prompting my score which I outline below. 

1. The first and major shortcoming of the paper is an extreme lack of detail and rigour. The paper seems to be written in a very hand wavy fashion. While that is okay in working towards the proposal of the algorithm, which of course is merely claimed to be inspired by parallel tempering. I believe the paper needs to do an important job of specifying exactly how one selects the notion of temperature for each choice of the hyperparameter. The mapping of the choice to the temperature value is essential for the algorithm to succeed and as described in the paper, there is no hope of replicating the algorithm as the authors are extremely vague about how to figure out this mapping. A second instance of this is diffusion curves in the introduction. The paper does not define what a numerical value of diffusion is. It only relies on a hand-waved understanding of diffusion to even understand presented curves. Note that the paper does not have an appendix and there is no link to code either. 

2. Lack of intuition - for this being a good procedure in terms of selecting a good path eventually. Note that hyperparameter optimization is an optimization procedure and it is very unclear to me why sampling from the distribution which will look like the product distribution over the replicas is even a good idea. If it is the case this needs to be specified carefully.
3. The experiments performed seem to follow a common trend. A single hyperparameter is selected (learning rate/dropout rate), a small grid of fixed values for the hyperparameter are selected and the procedure is performed. In light of this, what a “good” path through the algorithm represents is a good schedule for adjusting these hyperparameters. This is a well known intuition that for parameters like learning rate one needs to adjust them periodically to achieve good training dynamics. Since the comparison is to fixed learning rates, the improvement therefore looks predictable. 

4. Lack of applicability of the algorithm for hyperparameter search in practice - There are a couple of things which make the algorithm restrictive - 
This seems to be useful for hyperparameters that can be in a sense equated to parameters that change the training dynamics. The examples considered in the paper - batch size, learning rate, dropout rate. I think the authors should make this distinction and state it upfront because there are many hyperparameters that we optimize over which change the target distribution and not just the noise. 
Secondly, large scale hyperparameter search takes the idea of separating hyper parameter and parameter search as then it can run in a parallel asynchronous fashion. The algorithm is highly synchronous way of doing this search. While it can be argued that we should move to this paradigm if the results are significant but unfortunately no comparisons with existing hyperparameter search algorithms are provided. 

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This work presented an improvement of grid search algorithm for certain hyperparameters in deep neural nets training. These hyperparameters, such as learning rate and drop out, have "temperature" like meaning to control the noise injected in the training. With this analogy, the author proposed to use the idea of parallel tempering in statistical physics to allow exchange these hyperparameters during the training. Their empirical results showed this improves standard grid search.

The main contributions are two folds: 1. the connection between some hyperparameters to its physical perspective (temperature) and 2. the empirical validation of the proposed algorithm. The paper is clearly written and easy to follow. I would like to accept this paper if the authors can address following comments.

1. In Line 11 of algorithm 1, what is \alpha and how to update it?
2. How do the authors come up with the values for learning rate and drop out in the experiments? How sensitive the results are to the choice of those values?
3. What is the value of C used in the experiments. Do you have ablation study of this choice? How to pick this value?
review_title: Official Blind Review #2

Meta-review: This paper uses a variant of parallel tempering to tune the subset of neural net hyperparameters which control the amount of noise and/or rate of diffusion (e.g. learning rate, batch size). It's certainly an appealing idea to run multiple chains in parallel and periodically propose swaps between them. However, I'm not persuaded about the details. The argumentation in the paper is fairly informal, and it uses ideas from optimization and MCMC somewhat interchangeably. Since the individual chains aren't sampling from any known stationary distribution, it's not clear to me what MH-based swaps will achieve. 

The authors are upset with one of the reviews and think it misrepresents their paper. However, I find myself agreeing with most of the reviewer's points. Furthermore, as a general principle, the availability of code doesn't by itself make a paper reproducible. One should be able to reproduce it without the code, and one shouldn't need to refer to the code for important details about the algorithm.

Another limitation (pointed out by various reviewers) is that there aren't any comparisons against prior work on hyperparameter optimization. Overall, I think there are some promising and appealing ideas in this submission, but it needs to be cleaned up before it's ready for publication at ICLR.

Decision: Reject