Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper addresses multiple issues arising from the fact that commonly reported best model performance numbers are a single sample from a performance distribution. These problems are very real, and they deserve significant attention from the ML community.  However, I feel that the proposed solution may actually compound the issues highlighted.

Firstly, the proposed metric requires calculation of multiple test set experiments for every evaluation. In the paper up to 100 experiments were used. This may be reasonable in scenarios where the test set is hidden, and individual test numbers are never revealed. It also may be reasonable if we cynically assume that researchers are already running many test-set evaluations. But I am very opposed to any suggestion that we should relax the maxim that the test set should be used only once, or as close to once as is possible. Even the idea of researchers knowing their test set variance makes me very uneasy.

Secondly, this paper tries to account for variation in results due to different degrees of hyper-parameter tuning. This is certainly an admirable aim, since different research groups have access to very different types of resources. However, the suggested approach relies on randomly picking hyper-parameters from "a range that we previously found to work reasonably well". This randomization does not account for the many experiments that were required to find this range. And the randomization is also not extended to parameters controlling the model architecture (I suspect that a number of experiments went into picking the 32 layers in the ResNet used by this paper). Without a solid and consistent basis for these hyper-parameter perturbations, I worry that this approach will fail to normalize the effect of experiment numbers while also giving researchers an excuse to avoid reporting their experimental process.

I think this is a nice idea and the metric does merge the stability and low variance of mean score with the aspirations of best score. The metric may be very useful at development time in helping researchers build a reasonable expectation of test time performance in cases where the dev and test sets are strongly correlated. However, for the reasons outlined above, I don't think the proposed approach solves the problems that it addresses. Ultimately, the decision about this paper is a subjective one. Are we willing to increase the risk of inadvertent hyper-parameter tuning on the test set for the sake of a more stable metric?
review_title: This paper introduces a metric designed to address the inadequacy of best model performance numbers but, since this metric requires many runs on the test data and also claims to normalize different experimental processes without a solid basis for this normalization, it may actually compound the problems highlighted.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose a new measure to capture the inherent randomness of the performance of a neural net under different random initialisations and/or data inputs. Just reporting the best performance among many random realisations is clearly flawed yet still widely adopted. Instead, the authors propose to compute the so-called best-out-of-n performance, which is the expected best performance under n random initialisations. 

Pros:
- The widespread reporting of just the best model is clearly leading to very biased results and does not help with reproducibility. Any effort to mitigate this problem is thus welcome.
- The proposed quantity is simple to compute if we have m realisations of the same model under different random inputs (random initialisation or random data) and will converge to a stable limit even if m is very large. 

Cons:
- The best-out-of-n performance is well grounded if we have different random inputs such as random initial parameters or random batch processing. Arguably, there is even larger variance if the model parameters such as number of layers, layer size etc are varied. Yet these variations cannot really be captured by the best-out-of-n performance indicator unless modelled as random variables (which would lead to different sorts of problems).
- Computationally it requires to have a large number m of replications which is not always feasible. 
- Most importantly: the proposed way is just one of many ways to reduce the distribution of performances to a single scalar quantity. Why is it better than just reporting a specific quantile, for example? Perhaps any such attempt to reduce to a single scalar is flawed and we should report the full distribution (or first and second moment, or several quantiles). For example: the boo-n performance gets better if the outcome is highly variable compared to a model where the mean performance is identical but the outcome much less variable. High variance of the performance can be negative or positive, depending on the application and the choice of boo-n is making a singular choice just as if we chose the mean or min or max or a specific quantile. 













review_title: A scalar measure for architecture performance

final_score: rating: 6: Marginally above acceptance threshold
review_content: This manuscript raises an important issue regarding the current lack of standardization regarding methods for evaluating and reporting algorithm performance in deep learning research.  While I believe that raising this issue is important and that the method proposed is a step in the right direction, I have a number of concerns which I will list below.  One risk is that if the proposed solution is not adequate or widely agreeable then we may find a proliferation of solutions from which different groups might pick and choose as it suits their results!

The method of choosing the best model under 'internal' cross-validation to take through to 'external' cross-validation against a second hold-out set should be regarded as one possible stochastic solution to the optimisation problem of hyper-parameter selection.  The authors are right to emphasize that this should be considered part of the cost of the technique, but I would not suggest that one specify a 'benchmark' number of trials (n=5) for comparison.  Rather I would suggest that this is a decision that needs to be explored and understood by the researchers presenting the method in order to understand the cost/benefit ratio for their algorithm provided by attempting to refine their guess of the optimal hyperparameters.  This would then allow for other methods not based on internal cross-validation to be compared on a level footing.

I think that the fundamental issue of stochasticity of concern for repeatability and generalisability of these performance evaluation exercises is not in the stochastic optimisation search but in the use of a single hold-out sample.  Would it not be wise to insist on a mean performance (a mean Boo_n or other) over multiple random partitions of the entire dataset into training and hold-out?  I wonder if in theory both the effect of increasing n and the mean hold-out performance could be learnt efficiently with a clever experimental design. 

Finally, I am concerned with the issue of how to compute the suggested Boo_n score.  Use of a parameteric Gaussian approximation is a strong assumption, while bootstrap methods for order statistics can be rather noisy.  It would be interesting to see a comparison of the results from the parametric and non-parameteric Boo_n versions applied to the test problems.  
review_title: a step in the right direction

Meta-review: The subject of model evaluation will always be a contentious one, and the reviewers were not yet fully-convinced by the discussion. The points you bring up at the end of your rresponse already point to directions for improvement as well as a greater degree of precision and control.
Decision: Reject