Unit Tests for Stochastic Optimization
Authors: Tom Schaul, Ioannis Antonoglou, David Silver
ICLR 2014

Abstract: Optimization by stochastic gradient descent is an important component of many large-scale machine learning algorithms. A wide variety of such optimization algorithms have been devised; however, it is unclear whether these algorithms are robust and widely applicable across many different optimization landscapes. In this paper we develop a collection of unit tests for stochastic optimization. Each unit test rapidly evaluates an optimization algorithm on a small-scale, isolated, and well-understood difficulty, rather than in real-world scenarios where many such issues are entangled. Passing these unit tests is not sufficient, but absolutely necessary for any algorithms with claims to generality or robustness. We give initial quantitative and qualitative results on a dozen established algorithms. The testing framework is open-source, extensible, and easy to apply to new algorithms.


Peer review:
[Review]:
This paper bravely proposes to test the empirical convergence of stochastic optimization algorithms using a vast collection of simple and relatively standardized tests. They explain how they construct the tests and perform experiments that lead to a striking visualization (figure 5).  Unfortunately none of the compared algorithms appear to solve all the problems robustly. 

This idea could appear naïve because it is not supported by theoretical considerations and represents a purely empirical perspective. However there are many reasons to consider that this idea has great potential. First, similar ideas have worked in other fields. It is customary to compare general optimization codes on a collection of well known benchmark problems, not because it provides a guarantee, but because it provides a sanity check.  Second, we must recognize optimization in deep learning systems is still beyond the reach of theoretical analysis.  According to the current theoretical knowledge, it should not work. Therefore the best way to investigate such algorithms remains a well designed collection of empirical comparisons. The comparison described in this paper is a good start in that direction.