A Proposed Hierarchy of Deep Learning Tasks

1 INTRODUCTION . There are so many new exciting deep learning results published every day that it is hard to see the big picture . Many of these papers report exciting performance on a particular solution to a particular task , but it is hard to know how well those results are likely to generalize beyond the particulars in a particular paper . In undergraduate classes on Algorithms , we are taught how to reduce one problem to another , so we can make claims about time and space complexity that generalize across a wide range of problems . It would be much easier to make sense of the deep learning literature if we could find ways to generalize more effectively across problems . What can nets do , and what can ’ t they do ? Which tasks are relatively easy and which are relatively hard ? Which problems require more resources ? Is it possible to come up with an organization of the deep learning literature that is somewhat similar to the Chomsky Hierarchy , Chomsky ( 1957 ) , in Table 1 ? The Chomsky Hierarchy makes it clear what can ( and can not ) be done within broad classes of computational resources . Finite state machines can do many things , but they can ’ t solve the Halting Problem ( because nothing can solve the Halting Problem ) . Finite state machines also can ’ t solve problems that take more than constant space and linear time such as sorting large vectors , multiplying large matrices , and more . We would like to come up with statements about deep nets that generalize across topics . In particular , it might be useful to view certain exciting deep nets like convolutional neural nets as ( large ) finite state machines , and therefore , some of the statements mentioned above may apply to these nets . Obviously , there are well-known ways to go beyond finite-state , such as ( Sun et al . ( 2017b ) ) , but doing so is not necessarily a good thing ( because doing so tends to lead to increases in computational costs ) . Much of the excitement in neural nets is focused on recent progress in speech and vision , where it may not be necessary ( and even desirable ) to go beyond finite state . That said , it may not be all that helpful to focus too much on time and space , since training data appears to be more of a limiting factor on progress than time and space complexity . This paper will propose an organization of the deep nets literature based on training data , as opposed to time and space . The ranking of problems is intended to make it easier to generalize across tasks . If we have a new problem , can we compare it to a bunch of known problems in a meaningful way ? Can this ranking help us estimate how easy or hard the new problem is ? How do we estimate requirements on computational resources for a new problem ? This paper will suggest a particular proposal for ranking deep net problems based on learning curves . In general , there is no data like more data , but we find that some deep net tasks are more effective than others in taking advantage of more data . We suggest fitting learning curves to a power law , and then sorting tasks by empirical estimates of exponents . That is , we assume that loss , E ( m ) , can be modeled as a function of training set size , m , with : E ( m ) = αmβg . We propose sorting problems by the exponent : βg . Dimensionless quantities ( like βg ) are convenient for making comparisons across different problems . In general , different problems use different units . For example , in computational linguistics , it makes sense to measure the size of the training set in terms of words and/or characters , and to measure loss in terms of perplexity , but these metrics generally do not generalize well to other problems such as speech and vision . Since βg is dimensionless , it is more suitable for comparisons across problems than alternatives such as m , E ( m ) , α , where the units tend to vary from one problem to another . βg can depend on a number of factors including task and metric . We not only see important differences in βg across tasks , but we also see differences across metrics . As we will see in Section 4.1 , βg for top-1 classification can be different from βg for top-5 classification , even on the same task . It isn ’ t easy to measure βg over lots of solutions to lots of problems . It took about 50 GPU years to estimate the βgs in this paper . We not only considered a range of different problems , but also a range of different models ( with more or less capacity ) , optimizers , regularizers and loss functions . We will show examples where different solutions improve the constants , α , but we are more interested in improvements in βg . An interesting question for future work is why βg is different in different cases . Our estimates of βg are similar to estimates of cross-entropy . It is common practice with cross-entropy to try a number of different models , and return the best of those attempts . Cross-entropy is bounded by the true entropy of the task . There is a gap between the two because there may be a better solution than the ones that we considered . So too , with estimates of βg , it is hard to separate fundamental properties of the problems from concerns that there may be a better solution than the ones we considered . The main point of this paper is to start a discussion in search of a big picture that will eventually lead to agreement on how to generalize beyond the particulars in a particular deep net paper . It would be nice if our particular proposal is adopted , but it is more important to us that the field agree on a satisfactory solution than that they adopt our particular proposal . This paper is organized as follows : We begin with a review of related work measuring relationships between data set size , model size , and error in Section 2 . Section 3 describes our methodology , and Section 4 shows learning curve results from the five tasks . Section 5 applies our proposed organization to these tasks . 2 RELATED WORK . Empirical Studies : Figure 1 compares a well known result from ( Banko and Brill ( 2001 ) ) ( left ) with a cartoon sketch of a power law ( right ) . Banko and Brill ( 2001 ) test a language modeling problem ( confusion set disambiguation ) trained using subsets of a billion-word corpus of text . Their results show that there is no data like more data . Performance on this disambiguation task improves with the size of the training set , roughly consistent with a power law . They note that some learning methods are slightly better at one operating point , and others are slight better at another operating point . But those small differences are dominated by a more important trend . A rising tide lifts all boats ; more data improves performance for all methods . More data is more important than relatively small differences between methods . Power-law-like learning curves have been reported by a number of researchers for a number of applications . Jelinek refers to Banko and Brill ( 2001 ) and Lamel et al . ( 2002 ) in his speech accepting the Zampolli prize at LREC-2004.1 Amodei et al . ( 2016 ) show an improvement in WER ( word error rate ) for a Deep Speech 2 model with more and more training data ; Sun et al . ( 2017a ) report similar improvements for object detection and semantic segmentation tasks . The cartoon on the right of Figure 1 shows a similar power-law pattern for loss ( as opposed to accuracy ) . More training data improves performance ( left ) and reduces loss ( right ) . The cartoon on the right splits the learning curve into three regions . We are most interested in the middle region where the power-law assumption is more appropriate . The power-law assumption is less appropriate at the extremes where there is too much ( or too little ) training data . It is common in practice to have too little training data , but it is harder to find realistic examples with too much training data . In this study , we have been able to construct toy problems where the power law assumption breaks down with extremely large training set sizes , but we have failed to find that extreme case in real world examples in the wild . Theoretical Analysis : There is a considerable literature deriving theoretical bounds on the generalization gap between expected and empirical error . At least five lines of work derive bounds that follow power laws on the number of training samples : 1 . Hypothesis space bounds in PAC learning ( Haussler ( 1988 ) ) using Vapnik–Chervonenkis ( VC ) dimension ( Ehrenfeucht et al . ( 1989 ) ; Blumer et al . ( 1989 ) ; Haussler et al . ( 1996 ) ) , 2 . Rademacher complexity in ( Bartlett and Mendelson ( 2002 ) and Mohri et al . ( 2012 ) ) , 3. the stability approach in ( Bousquet and Elisseeff ( 2002 ) ) considers the dependence of the learned model on the training dataset , as well as how much individual changes to the training dataset can affect the learned model , 1http : //www.lrec-conf.org/lrec2004/doc/jelinek.pdf 4. the robustness approach in ( Xu and Mannor ( 2012 ) ) measures how much the error can vary with the input space , 5 . ( Amari et al . ( 1992 ) and Amari ( 1993 ) ) derive bounds using statistical mechanics . Kawaguchi et al . ( 2017 ) provides a clear survey of much of this prior work . We follow the tradition in much of this literature of introducing a power law assumption . In spite of the complex nature of real tasks , the reliable observation of power-law learning curves in our experiments suggests that deep learning exhibits the same scaling phenomena as those that are simple to analyze theoretically . Model Capacity Required to Fit Data : This work will assume that both loss and model capacity obey power laws . Prior studies propose various measures of model capacity based on a model ’ s organization and parameterization , and these measures hint at the model size required to fit a training set . Vapnik and Chervonenkis defined the VC dimension of a model as the cardinality of the largest set of data points that a model can shatter ( Vapnik ( 1998 ) ) . Follow-on work uses data complexity measures to estimate the structure of model families that might fit the data ( Bartlett and Mendelson ( 2002 ) ) . Recent work explores applying these concepts to deep nets ( Harvey et al . ( 2017 ) ; Dziugaite and Roy ( 2017 ) ; Collins et al . ( 2017 ) ) . Prior work to empirically estimate model scaling with training set size is very sparse . In our experiments , we find power-laws produce good fits , i.e . s ( m ) ∝ αmβp , where s ( m ) is the required model size to fit a training set of size m , and βp ∈ [ 0.5 , 1 ] . When applying this approach to the data in Banko and Brill ( 2001 ) , we find that the Winnow and memory-based models grow with the same power-law exponent to larger data sets , βp ≈ 0.72 .

Summary: The authors propose to measure the power-law exponent to sort natural language processing, speech and vision problems by the degree of their difficulty. The main idea is that, while in general model performance goes up for most tasks if more training data becomes available or for bigger model sizes, some tasks are more effective at leveraging more data. Those tasks are supposed to be easier on the proposed scale.