A Proposed Hierarchy of Deep Learning Tasks

1 INTRODUCTION . There are so many new exciting deep learning results published every day that it is hard to see the big picture . Many of these papers report exciting performance on a particular solution to a particular task , but it is hard to know how well those results are likely to generalize beyond the particulars in a particular paper . In undergraduate classes on Algorithms , we are taught how to reduce one problem to another , so we can make claims about time and space complexity that generalize across a wide range of problems . It would be much easier to make sense of the deep learning literature if we could find ways to generalize more effectively across problems . What can nets do , and what can ’ t they do ? Which tasks are relatively easy and which are relatively hard ? Which problems require more resources ? Is it possible to come up with an organization of the deep learning literature that is somewhat similar to the Chomsky Hierarchy , Chomsky ( 1957 ) , in Table 1 ? The Chomsky Hierarchy makes it clear what can ( and can not ) be done within broad classes of computational resources . Finite state machines can do many things , but they can ’ t solve the Halting Problem ( because nothing can solve the Halting Problem ) . Finite state machines also can ’ t solve problems that take more than constant space and linear time such as sorting large vectors , multiplying large matrices , and more . We would like to come up with statements about deep nets that generalize across topics . In particular , it might be useful to view certain exciting deep nets like convolutional neural nets as ( large ) finite state machines , and therefore , some of the statements mentioned above may apply to these nets . Obviously , there are well-known ways to go beyond finite-state , such as ( Sun et al . ( 2017b ) ) , but doing so is not necessarily a good thing ( because doing so tends to lead to increases in computational costs ) . Much of the excitement in neural nets is focused on recent progress in speech and vision , where it may not be necessary ( and even desirable ) to go beyond finite state . That said , it may not be all that helpful to focus too much on time and space , since training data appears to be more of a limiting factor on progress than time and space complexity . This paper will propose an organization of the deep nets literature based on training data , as opposed to time and space . The ranking of problems is intended to make it easier to generalize across tasks . If we have a new problem , can we compare it to a bunch of known problems in a meaningful way ? Can this ranking help us estimate how easy or hard the new problem is ? How do we estimate requirements on computational resources for a new problem ? This paper will suggest a particular proposal for ranking deep net problems based on learning curves . In general , there is no data like more data , but we find that some deep net tasks are more effective than others in taking advantage of more data . We suggest fitting learning curves to a power law , and then sorting tasks by empirical estimates of exponents . That is , we assume that loss , E ( m ) , can be modeled as a function of training set size , m , with : E ( m ) = αmβg . We propose sorting problems by the exponent : βg . Dimensionless quantities ( like βg ) are convenient for making comparisons across different problems . In general , different problems use different units . For example , in computational linguistics , it makes sense to measure the size of the training set in terms of words and/or characters , and to measure loss in terms of perplexity , but these metrics generally do not generalize well to other problems such as speech and vision . Since βg is dimensionless , it is more suitable for comparisons across problems than alternatives such as m , E ( m ) , α , where the units tend to vary from one problem to another . βg can depend on a number of factors including task and metric . We not only see important differences in βg across tasks , but we also see differences across metrics . As we will see in Section 4.1 , βg for top-1 classification can be different from βg for top-5 classification , even on the same task . It isn ’ t easy to measure βg over lots of solutions to lots of problems . It took about 50 GPU years to estimate the βgs in this paper . We not only considered a range of different problems , but also a range of different models ( with more or less capacity ) , optimizers , regularizers and loss functions . We will show examples where different solutions improve the constants , α , but we are more interested in improvements in βg . An interesting question for future work is why βg is different in different cases . Our estimates of βg are similar to estimates of cross-entropy . It is common practice with cross-entropy to try a number of different models , and return the best of those attempts . Cross-entropy is bounded by the true entropy of the task . There is a gap between the two because there may be a better solution than the ones that we considered . So too , with estimates of βg , it is hard to separate fundamental properties of the problems from concerns that there may be a better solution than the ones we considered . The main point of this paper is to start a discussion in search of a big picture that will eventually lead to agreement on how to generalize beyond the particulars in a particular deep net paper . It would be nice if our particular proposal is adopted , but it is more important to us that the field agree on a satisfactory solution than that they adopt our particular proposal . This paper is organized as follows : We begin with a review of related work measuring relationships between data set size , model size , and error in Section 2 . Section 3 describes our methodology , and Section 4 shows learning curve results from the five tasks . Section 5 applies our proposed organization to these tasks . 2 RELATED WORK . Empirical Studies : Figure 1 compares a well known result from ( Banko and Brill ( 2001 ) ) ( left ) with a cartoon sketch of a power law ( right ) . Banko and Brill ( 2001 ) test a language modeling problem ( confusion set disambiguation ) trained using subsets of a billion-word corpus of text . Their results show that there is no data like more data . Performance on this disambiguation task improves with the size of the training set , roughly consistent with a power law . They note that some learning methods are slightly better at one operating point , and others are slight better at another operating point . But those small differences are dominated by a more important trend . A rising tide lifts all boats ; more data improves performance for all methods . More data is more important than relatively small differences between methods . Power-law-like learning curves have been reported by a number of researchers for a number of applications . Jelinek refers to Banko and Brill ( 2001 ) and Lamel et al . ( 2002 ) in his speech accepting the Zampolli prize at LREC-2004.1 Amodei et al . ( 2016 ) show an improvement in WER ( word error rate ) for a Deep Speech 2 model with more and more training data ; Sun et al . ( 2017a ) report similar improvements for object detection and semantic segmentation tasks . The cartoon on the right of Figure 1 shows a similar power-law pattern for loss ( as opposed to accuracy ) . More training data improves performance ( left ) and reduces loss ( right ) . The cartoon on the right splits the learning curve into three regions . We are most interested in the middle region where the power-law assumption is more appropriate . The power-law assumption is less appropriate at the extremes where there is too much ( or too little ) training data . It is common in practice to have too little training data , but it is harder to find realistic examples with too much training data . In this study , we have been able to construct toy problems where the power law assumption breaks down with extremely large training set sizes , but we have failed to find that extreme case in real world examples in the wild . Theoretical Analysis : There is a considerable literature deriving theoretical bounds on the generalization gap between expected and empirical error . At least five lines of work derive bounds that follow power laws on the number of training samples : 1 . Hypothesis space bounds in PAC learning ( Haussler ( 1988 ) ) using Vapnik–Chervonenkis ( VC ) dimension ( Ehrenfeucht et al . ( 1989 ) ; Blumer et al . ( 1989 ) ; Haussler et al . ( 1996 ) ) , 2 . Rademacher complexity in ( Bartlett and Mendelson ( 2002 ) and Mohri et al . ( 2012 ) ) , 3. the stability approach in ( Bousquet and Elisseeff ( 2002 ) ) considers the dependence of the learned model on the training dataset , as well as how much individual changes to the training dataset can affect the learned model , 1http : //www.lrec-conf.org/lrec2004/doc/jelinek.pdf 4. the robustness approach in ( Xu and Mannor ( 2012 ) ) measures how much the error can vary with the input space , 5 . ( Amari et al . ( 1992 ) and Amari ( 1993 ) ) derive bounds using statistical mechanics . Kawaguchi et al . ( 2017 ) provides a clear survey of much of this prior work . We follow the tradition in much of this literature of introducing a power law assumption . In spite of the complex nature of real tasks , the reliable observation of power-law learning curves in our experiments suggests that deep learning exhibits the same scaling phenomena as those that are simple to analyze theoretically . Model Capacity Required to Fit Data : This work will assume that both loss and model capacity obey power laws . Prior studies propose various measures of model capacity based on a model ’ s organization and parameterization , and these measures hint at the model size required to fit a training set . Vapnik and Chervonenkis defined the VC dimension of a model as the cardinality of the largest set of data points that a model can shatter ( Vapnik ( 1998 ) ) . Follow-on work uses data complexity measures to estimate the structure of model families that might fit the data ( Bartlett and Mendelson ( 2002 ) ) . Recent work explores applying these concepts to deep nets ( Harvey et al . ( 2017 ) ; Dziugaite and Roy ( 2017 ) ; Collins et al . ( 2017 ) ) . Prior work to empirically estimate model scaling with training set size is very sparse . In our experiments , we find power-laws produce good fits , i.e . s ( m ) ∝ αmβp , where s ( m ) is the required model size to fit a training set of size m , and βp ∈ [ 0.5 , 1 ] . When applying this approach to the data in Banko and Brill ( 2001 ) , we find that the Winnow and memory-based models grow with the same power-law exponent to larger data sets , βp ≈ 0.72 .

Summary: The paper proposes an empirical solution to coming up with a hierarchy of deep learning tasks or in general machine learning tasks. They propose a two-way analysis where power-law relations are assumed between (a) validation loss and training set size, and (b) the number of parameters of the best model and training set size. The first power-law exponent, \beta_g, indicates how much can more training data be helpful for a given task and is used for ordering the hardness of problems. The second power-law exponent, \beta_p, indicates how effectively does the model use extra parameters with increasing training set (can also be thought of as how good the model is at compression). From experiments across a range of domains, the authors find that indeed on tasks where much of the progress has been made tend to be ones with smaller \beta_g (and \beta_p). It's arguable as to how comparable these power-law exponents are across domains because of differences in losses and other factors, but it's definitely a good heuristic to start working in this direction.