Unit Tests for Stochastic Optimization
Authors: Tom Schaul, Ioannis Antonoglou, David Silver
ICLR 2014

Abstract: Optimization by stochastic gradient descent is an important component of many large-scale machine learning algorithms. A wide variety of such optimization algorithms have been devised; however, it is unclear whether these algorithms are robust and widely applicable across many different optimization landscapes. In this paper we develop a collection of unit tests for stochastic optimization. Each unit test rapidly evaluates an optimization algorithm on a small-scale, isolated, and well-understood difficulty, rather than in real-world scenarios where many such issues are entangled. Passing these unit tests is not sufficient, but absolutely necessary for any algorithms with claims to generality or robustness. We give initial quantitative and qualitative results on a dozen established algorithms. The testing framework is open-source, extensible, and easy to apply to new algorithms.


Peer review:
[Review]:
Summary
------------

This paper introduces a suite of unit tests for optimization algorithms that attempt to analyze the algorithm performance in very specific and isolated cases (for example : how does it deal with a saddle point or a cliff like structure in the error). This work feels like a needed step in the right direction. As the authors pointed out, passing these tests is not sufficient to claim an algorithm is better than another, but it is necessary to understand some possible weaknesses of the algorithm and for doing a more proper comparison between different algorithms (compared to just looking at the error curve for some randomly selected task and model).
Comments:
--------------

I think the success of such an approach is in the details. The engineering effort to use a specific such framework to test some new algorithm will have a huge impact on whether these unit tests will be successful or not. Unfortunately some of these things are hard to predict right now (and in some sense are beside the point of the paper).
The other thing that one has to keep in mind is the effect of high dimensional spaces on the problem we try to solve. E.g. one can prove that for some family of models (say auto encoders) on some bounded domain some given error will have saddle points regardless of the dimensionality of the input. However the distribution of this saddle points is a different story and it might be hard to say how important it is to deal with saddle points or not for this specific model.

Basically what am I trying to say is that the existence of some prototype in real data is easy to assert, but the probability that you will have to deal with said prototype is hard. And this is not a criticism to this work, but more a message that anyone should keep in mind. I guess what I'm trying to point out is that interpreting the results of these unit tests is far from trivial and might be even counter-intuitive.

Another important detail for any such suite of unit tests is the tools that one can use to investigate the results. If I think of the unit test metaphor for code development, usually the results of a large suite of tests is interpreted via failure (and how many tests failed) or success. Running such a suite of experiments on an optimization task is not as easy to interpret. First of all is hard to know which tests we care most about, and is hard to know how the correlation between the failures on different tests affect the algorithm on real task. While the paper mentions that there are tools for visualizing these results, there are not many details given about these tools. I think an interesting question on its own is what are the write visualization of the results and how to interpret them, a question which I don't think is fully answer by the current work.