Title: Universality in halting time
Authors: Levent Sagun
Thomas Trogdon
Yann LeCun
Venue: ICLR 2017
Keywords: Optimization

Abstract: The authors present empirical distributions for the halting time (measured by the number of iterations to reach a given accuracy) of optimization algorithms applied to two random systems: spin glasses and deep learning. Given an algorithm, which we take to be both the optimization routine and the form of the random landscape, the fluctuations of the halting time follow a distribution that remains unchanged even when the input is changed drastically. We observe two main classes, a Gumbel-like distribution that appears in Google searches, human decision times, QR factorization and spin glasses, and a Gaussian-like distribution that appears in conjugate gradient method, deep network with MNIST input data and deep network with random input data. This empirical evidence suggests presence of a class of distributions for which the halting time is independent of the underlying distribution under some conditions.

TL;DR: Normalized halting time distributions are independent of the input data distribution.

Review discussion:
What you do in the paper is quite clear however i do not fully understand your motivations. In a spirit of fairness, I prefer to candidly ask difficult questions now that you have the time to answer them.
Why should I care about the distribution halting time and the universal property ? How does this relate (in a useful way) to deep networks, learning representations ?
In the conclusion you write:
"What constitutes a good set of hyperparameters for a given algorithm? [...] How can we infer if an algorithm is a good match to the system at hand? [NDR: two questions I relate to]
You then write:
"This research attempts to exhibit cases where one can extract answers to these questions in a robust and quantitative way"
However I do not find where this paper answers (or attempts to answer) these questions. Could you point me to the right pages/paragraphs ?
A short answer would be to be able to estimate the time it takes for an algorithm to converge at a given accuracy: Once the shape is known, one would collect small number of samples to estimate the mean and the standard deviation of the halting time for an approximation.
Such an approximation may have both applied and theoretical consequences: On the one hand, a lot of theoretical research typically focus on the error bounds as a function of accuracy, step-size, number of steps and possibly other parameters with possible asymptotical tightness. It may well be possible that such bounds are much further away from the right tail of the distributions we find much like the comparison between exact Chebyshev bounds for the worst case and CLT approximations for typical behaviors. We believe that such an approximation would be at least as valuable as tight-asymptotic bounds.
From an applied point of view, the typical behaviour may be an indicative of robustness for the system depending on wheter the halting time distribution falls into the desired universal class or not which may be tested with random data on a smaller model. With this in mind, the question of the hyperparameter choice, and algorithm-system match can be interpreted as the inverse questions to what we present in our work.
For instance, setting a sharper stopping criteria might result in the loss of universality. It can be seen in figure 6 of the appendix, that the right side with sharper stopping criteria corresponds to a region where the training time increases by quite a lot without any meaningful returns in the performance of the system. If we are able to determine the universal regime as a function of such parameters, or if we can experimentally identify such a region, then we would be able to assign more principled targets for the given algorithm-system.
Thank you for bringing this question to our attention for further emphasis. We added more clarification of the points above in our conclusion section and at the beginning of our appendix inculing further references. And we would like to pursue similar explorations (possibly in collaboration with interested readers) of this inverse question as part of the implications of our current work in the near future.
In Figure 1, what does the x-axis represent? The term "search time fluctuations" is a little confusing.
It should have been the halting time fluctuations just like the x-axis for other figures in the paper. The halting times, in the case of figure 1, are obtained by normalizing the measured time it takes to get an answer for the two experiments described: Google searches and questions asked on subjects. We fixed the x-axis and will include in the updated pdf.
The idea of universality that is independent of input distribution and dimension, depending only on the algorithm is an appealing one.
However, as an empirical study, this paper comes up somewhat short:
1. Exactly one algorithm is shown for the deep learning example. It would have been more convincing to compare distributions with one or more algorithms.
2. The definition (1),  and much of the work of Section 2.1 seems to have already been covered in Deift (2014), Section 1.3. In that paper, a number of different algorithms for the solution of linear systems are considered, and then the concept of universality becomes more plausible. I do not see enough of such algorithmic comparisons in this paper (same problem setup, different algorithms).
3.  It seems to me that what practitioners might care about in practice are both the mean and variance in running times; these quantities are buried in (1).
So I question how useful the distribution itself might be for algorithm tuning.
At the least, many more empirical comparisons should be provided to convince me that the universality holds across a broad range of algorithms.
In this paper we largely focus on one widely used method: (stochastic) gradient descent.  We do apply this to 4 different random landscapes.  We consider the method and the random landscape, combined, to constitute an algorithm.   This adds to the convincing empirical and rigorous evidence obtained by Deift et al. that universality is a persistent phenomenon in numerical computation. We also present the example of Google searches --- yet another piece of evidence.
For this reason we believe the current work is sufficiently comprehensive.
But we do concede that there is much more work to be done in these directions, both rigorous and experimental.
We agree that the mean and the variance are likely the most important in practice.
When considering the Central Limit Theorem, one knows that the mean and the variance are the only required pieces of information precisely because the theorem gives a universal limit.  This is our point here: To consistently argue that the mean and the variance are the most important, one wants a universal limit theorem.
Summary
For several algorithms, previous research has shown that the halting time follows a two-parameter distribution (the so-called universal property investigated by the authors). In this work, the authors extend the investigation to new algorithms (spin-glass, gradient descent in deep learning).
An algorithm is considered to satisfy the universality property when the centered/scaled halting time fluctuations (empirical distribution of halting times) depend on the algorithm but do not depend on the target accuracy epsilon, an intrinsic measure of dimension N, the probability distribution/random ensemble. (This is clear from Eq 1 where on the left the empirical halting time distribution depends on epsilon, N, A, E and on the right, the approximation only depends on the algorithm)
The authors argue that empirically, the universal property is observed when both algorithms (spin glass and deep learning) perform well and that it is not observed when they do not perform well.
A moment-based indicator is introduced to assess whether universality is observed.
Review
This paper presents several problems.
page 2: “[…] for sufficiently large N and eps = eps(N)”
The dependence of epsilon on N is troubling.
page 3: “Universality is a measure of stability in an algorithm […] For example […] halting time for the power method […] has infinite expectation and hence this type of universality is *not* present. One could use this to conclude that the power method is naive.
Therefore the presence of universality is a desirable feature of a numerical method”
No. An algorithm is naive if there are better ways to answer the problem. One could not conclude from a halting time with infinite expectation (e.g. solving a problem extremely quickly 99% of the time, and looping forever in 1% of cases) or infinite variance, that the algorithm is naive.
Moreover the universal property is more restrictive than having a finite halting time expectation.
Even if in many specific cases, having a finite halting time expectation is a desirable property, showing that the presence of universality is desirable would require a demonstration that the other more restrictive aspects are also desirable.
Also, the paragraph only concerns one algorithm. why would the conclusions generalise to all numerical methods ?
Even if the universality property is arguably desirable (i.e. event if the conclusion of this paragraph is assumed correct), the paragraph does not support the given conclusion.
Comparing Eq 1 and figures 2,3,4,5
From Eq 1, universality means that the centered/scaled halting time fluctuations (which depend on A, epsilon, N, E) can be approximated by a distribution that only depends on A (not on epsilon, N, E) but in the experiments only E varies (figures 2,3,4,5). The validity of the approximation with varying epsilon or N is never tested
The ensembles/distributions parameter E (on which halting fluctuations should not depend) and the algorithm A (on which halting fluctuations are allowed to depend) are not well defined, especially w.r.t. the common use of the words. In the optimisation setting we are told that the functional form of the landscape function is part of A (in answer to the question of a reviewer) but what is part of the functional form ? what about computations where the landscape has no known functional form (black box) ?
The conclusion claims that the paper “attempts to exhibit cases” where one can answer 5 questions in a robust and quantitative way.
Question 1: “What are the conditions on the ensembles and the model that lead to such universality ?”
The only quantitative way would be to use the moments based indicator however there is only one example of universality not being observed which concerns only one algorithm (conjugate gradient) and one type of failure (when M = N). This does not demonstrate robustness of the method.
Question 2: “What constitutes a good set of hyper parameters for a given algorithm ?”
The proposed way to choose would be to test whether universality is observed. If it is then the hyper parameters are good, if not the hyper parameters are bad. The correspondance between bad hyper-parameters and observing no universality concerns only one algorithm and one type of failure. Other algorithms may fail in the universal regime or perform well in the non universal regime. The paper does not show how to answer this question in a robust way.
Question 3: "How can we go beyond inspection when tuning a system ?
The question is too vague and general and there is probably no robust and quantitative way to answer it at all.
Question 4: "How can we infer if an algorithm is a good match to the system at hand ?
The paper fails to demonstrate convincingly that universality is either a good or robust way to approach the very few studied algorithms. The suggested generalisation to all systems and algorithms is extremely far fetched.
Question 5: "What is the connection between the universal regime and the structure of the landscape ?"
Same as before, the question is extremely vague and cannot be answered in a robust or quantitative way at all. The fact that what corresponds to A and what corresponds to E is not clear does not help.
In the conclusion it is written that the paper validates the claim that universality is present in all or nearly all sensible computation. It does not. The paper does not properly test whether universality is present (only 1 parameter in 3 that should not vary is tested). The paper does not properly test whether universality is lost when the computation is no longer sensible (only one failure case tested).
Finally the experiments do not apply to all or nearly all computations but only to very few  specific algorithms.
We appreciate the detailed look on the paper and we strongly dispute the charge that our methods are unsound. This paper aims solely to present experimentally observed phenomena that machine learning computations exhibit. We address the specific concerns below.
Regarding the epsilon dependency:
In the recent paper of Deift & Trogdon (2017, arXiv:1701.01896) the authors prove universality for the time to compute the top eigenvalue of a matrix. With the chosen scaling, the gap between the top eigenvalue and the second-largest eigenvalue shrinks with N. And so, epsilon must shrink with N so that one can be sure the top eigenvalue is the one being computed.
But in practice, epsilon = 10^(-14) will suffice for all matrices encountered.
In short, the N dependence is needed for theoretical justification and is often an artifact of the scaling of the problem, and importantly, the dependence could be trivial.
Regarding naive algorithms:
Clearly, an algorithm is naive if there is a better algorithm.
But, such an over-simplified characterization may not be of any use in computation: Given an algorithm, how does one know it is naive? Is a naive algorithm still faster on some problems than a more sophisticated algorithm? Is it comparable in a statistical sense? The phenomenon that we discuss was shown by Deift et al. to persist in numerous algorithms. Our work aims to demonstrate that this phenomenon exists in machine learning and optimization (in 4 settings, with additional compelling data from Google searches).
Regarding Eq 1 and figures:
In our computations we chose N large and epsilon small.
It is true that epsilon and N should vary and we did do these computations and we have plots for them in the appendix. We will add their halting time histograms, as well.
Regarding the questions raised:
The referee raises many questions, and points out that we have not answered questions we raise in full detail. We accept this criticism. The primary goal of this work was to demonstrate the universality in the halting time can occur within machine learning and to propose potential implications. There are many open problems related to this line of research and we do not claim that the book is closed on such matters.
The problem right now with your conclusion is not that you have not answered all questions in full detail in your paper. The problem is that you pretty much claim that you did.
Your comment suggests that you are aware that the contribution is less important that what is portrayed in the conclusion.
In fact you do not present any argument to show that your conclusion is accurate as it is.
- If an accurate summary of your work is (as in your above comment):
A: "The primary goal of this work was to demonstrate the universality in the halting time can occur within machine learning and to propose potential implications."
- Then what is written right now in your conclusion is not:
B: "This research attempts to exhibit cases where one can extract **answers to these questions in a robust and quantitative way**."
and
C: "It also **validates the broad claims** made in Deift et al. (2015) that **universality is present in all or nearly all (sensible) computation**
The differences are striking:
"answers in a robust and quantitative way" (in your paper) VS "propose potential implications" (in your above comment)
"universality is present  in nearly all computation" (in your paper) VS "universality [...] *can occur*" (in your above comment)
The authors explore whether the halting time distributions for various algorithms in various settings exhibit "universality", i.e. after rescaling to zero mean and unit variance, the distribution does not depend on stopping parameter, dimensionality and ensemble.
The idea of the described universality is very interesting.
However I see several shortcomings in the paper:
In order to be of practical relevance, the actual stopping time might be more relevant than the scaled one. The discussion of exponential tailed halting time distributions is a good start, but I am not sure how often this might be actually helpful.
Still, the findings in the paper might be interesting from a theoretical point of view.
Especially for ICLR, I think it would have been more interesting to look into comparisons between stochastic gradient descent, momentum, ADAM etc on different deep learning architectures. Over which of those parameters does universality hold?. How can different initializations influence the halting time distribution? I would expect a sensible initialization to cut of part of the right tail of the distribution.
Additionally, I found the paper quite hard to read. Here are some clarity issues:
- abstract: "even when the input is changed drastically": From the abstract I'm not sure what "input" refers to, here
- I. Introduction: "where the stopping condition is, essentially, the time to find the minimum": this doesn't seem to make sense, a condition is not a time. I guess the authors wanted to say that the stopping condition is that the minimum has been reached?
- I.1 the notions of dimension N, epsilon and ensemble E are introduced without any clarification what they are. From the later parts of the paper I got some ideas and examples, but here it is very hard to understand what these parameters should be (just some examples would be already helpful)
- I.3 "We use x^\ell for \ell \in Z=\{1, \dots, S\} where Z is a random sample from of training samples" This formulation doesn't make sense. Either Z is a random sample, or Z={1, ..., S}.
- II.1 it took me a long time to find the meaning of M. As this parameter seems to be crucial for universality in this case, it would be very helpful to point out more explicitly what it refers to.
We agree that one would like to understand the true halting time.
But, if the halting time is universal (or nearly so), after normalizing to mean zero and variance one, it follows that the mean and the variance of the actual halting time are the relevant quantities: Higher moments are not important.
We appreciate the ICLR specific suggestions. They are certainly relevant experiments to perform. In our experience, this type of universality, being different in spirit than a lot of work in the community, can be difficult for the reader to digest. To attempt to remedy this issue, we decided to focus on stochastic gradient descent, giving plenty of space for an introduction. In the continuation of this work, we will certainly make an effort in the direction of these suggestions.
Regarding the clarity (in the same order):
- We changed this sentence to read: "... follow a distribution that, after centering and scaling, remains unchanged even when the distribution on the landscape is changed."
- We changed this sentence to read: "... the stopping condition produces a halting time that is, essentially, the time to find the minimum."  This should be contrasted with the non-convex case where the true minimum is never found.
- Section 1.2 is devoted to an example that was initially examined in Pfrang, Deift & Menon (2014).  We added a sentence, that points forward to this section, highlighting the meaning of the parameter:
"For example, in Section 1.2, A is the QR eigenvalue algorithm, B is the size of the matrix, epsilon is a small tolerance and E is given by a distribution on complex Hermitian (or real symmetric) matrices."
- We modified this to read:  Here $x^{\ell} \in Z$ for $\ell \in  \{1,...,S\}$, where $Z$ is a random (ordered) sample of size $S$ from the training examples.
- We added the following sentence near where $M$ is introduced, "The choice of the integer $M$, which is the inner dimension of the matrices in the product $XX^*$, is critical for the existence of universality."

Meta-review:
role: PC
data: comment: My overall conclusion is that the paper needs more work to be sufficiently convincing. I think the reviews are sufficiently careful. My recommendation is based on the fact that none of the reviewers supports acceptance.

Decision: Reject