Top-down training for neural networks

1 INTRODUCTION . Deep neural networks ( DNNs ) have given impressive results in many domains , such as speech recognition ( Hinton et al. , 2012 ) , natural language processing ( Bahdanau et al. , 2014 ) and computer vision ( Krizhevsky et al. , 2012 ) ; however , gradient-based training of DNNs remains a non-trivial task , owing to the vanishing gradient problem ( Bengio et al. , 1994 ; Hochreiter , 1998 ) which arises from the deep model structure . The vanishing gradient problem does not only make the training slow , it can also result in the bottom layers closer to the input being under-trained , while over-training the top layers closer to the output . We can interpret the bottom layers of the neural network as the feature extractor , and the top layers as the classifier . During back-propagation , the gradient is decided by the current values of the parameters and all layers are trained jointly . Thus , the feature extractor is learning to provide features which fit the partition of the space defined by the current classifier , while the classifier is learning to fit the features generated by the current feature extractor . Ideally , the speed of learning for the feature extractor and the classifier should be similar . However , since the partial derivatives of the top layers are much larger than the partial derivatives of the bottom layers , the classifier learns to fit to the feature extractor much faster than the feature extractor learns to fit to the classifier . Thus , the feature extractor layers are usually “ under-trained ” ( they underfit the classifier ) while the classifier are in general “ over-trained ” ( they overfit the feature extractor ) . Also , since the feature extractor learns more slowly , then potentially the classifier may overfit the feature extractor before the feature extractor is able to learn much about the underlying pattern of the data . Pre-training techniques can alleviate this problem ( Hinton et al. , 2006 ; Bengio et al. , 2007 ; Larochelle et al. , 2009 ; Erhan et al. , 2010 ) . In these methods , each layer is pre-trained to learn the key patterns of the distribution of outputs of the previous layer . Thus , the feature extractors are initialized in a region where they have already captured relevant information . Another method is using “ short-cut ” or “ highway ” connections ( He et al. , 2016 ; Huang et al. , 2017 ; Srivastava et al. , 2015 ) . In these methods , the feature extractor and classifier are updated in a more synchronized manner . Moreover , an auxiliary loss ( Szegedy et al. , 2015 ; Lu et al. , 2019 ) can be added to the intermediate layers to bring the training signal closer to them and the layers below . Lu et al . ( 2019 ) show that for speech recognition tasks , even with 30,000 hours of training data , the training still suffers from the vanishing gradient problem . In this paper , we propose to alleviate the vanishing gradient problem in a top-down manner . Instead of pre-training a feature extractor bottom-up layer by layer , we first train a good classifier , then freeze it , and then train the bottom layers with respect to the classifier . To be more precise , we denote the loss function by L , the function defined by the frozen top layers by T and the composition of these two functions by L ◦ T . Training the bottom layers is equivalent to training a shallower network with the loss function L ◦ T . Although the partial derivatives of the bottom layers may be small since ∂L/∂T can be small , this will only lead to slow training , not to under-fitting the feature extractor ; this is because the classifier can not be updated to overfit the outputs of the feature extractor or stop the training . Instead , the training terminates only when the feature extractor has fitted the current frozen classifier . Thus , the feature extractor can not be left in an under-trained state . Moreover , the well-trained feature extractor has the potential of making the model generalize better . We summarize this argument as the good classifier hypothesis . The good classifier hypothesis . We interpret the top layers of a neural network as a classifier and the bottom layers as a feature extractor . The vanishing gradient problem leads to the classifier being over-trained and the feature extractor under-trained . The under-trained feature extractor tends to capture the underlying patterns in the data less well . However , When trained with a frozen classifier , the feature extractor can fit to the classifier and becomes well-trained . If the classifier is good enough – that is , if it divides the space in a way such that the features that fit the partition reflect more about the essential patterns of the data – then the well-trained feature extractor is able to capture these patterns . As a result , the model will generalize better . In this paper , we present a novel top-down training method based on the good classifier hypothesis , which we investigate with a set of speech recognition experiments . In these experiments we observe that the top-down approach leads to better generalization compared to conventional joint training . To our knowledge , this is the first time that this hypothesis is proposed . Also , to our knowledge , this is the first time that supervised top-down training is proposed and significant performance gains observed . Our findings have the potential to guide the development of new training algorithms and new transfer learning methods . 2 TRAINING FEATURE EXTRACTOR BASED ON A GOOD CLASSIFIER . In this section , we empirically justify the good classifier hypothesis . Before we give grounds for the hypothesis , we define the notion of a “ good classifier ” . Here , the word “ good ” is a relative term . To better illustrate what is a good classifier , we first define a cut of a neural network . A cut of a neural network . For a n-layer neural network , a cut Ci is a partition of the layers l = ( l1 , . . . , li , li+1 , . . . , ln ) into two ordered sequences of layers b = ( l1 , . . . , li ) and t = ( li+1 , . . . , ln ) , where l1 denotes the bottom-most layer and ln denotes the top-most layer . With this definition , we define a good classifier . A good classifier for a neural network . For a n-layer neural network , we apply a cutCi to it . To the top layers t we assign a set of parameters and denote it as ta . We assign t another set of parameters and denote it as tb . If the features fit to ta contain more information about the data pattern than the features fit to tb , then ta is a good classifier relative to tb . By the good classifier hypothesis , the model trained with the frozen classifier ta should generalize better than the model trained with the frozen classifier tb , since ta will force the feature extractor to learn more about the underlying pattern . To verify this , we have conducted a series of end-toend speech recognition experiments using the Wall Street Journal ( WSJ : Paul & Baker , 1992 ) and CHiME-4 ( Vincent et al. , 2017 ; Barker et al. , 2017 ) corpora . We train a network on the WSJ si284 dataset , a large vocabulary clean speech dataset . We construct a subset ( 25 % size ) of the si284 dataset by randomly sampling utterances from si284 . Then we train a model on the si284 subset . We also train a model using CHiME-4 tr05 multi , a dataset containing WSJ utterances with real and simulated noise . We assume the top layers of the model trained on the full WSJ si284 set is a good classifier , while the top layers of the models trained on si284 subset or on tr05 multi are poorer classifiers – since it is easier for a network trained on a relatively large , clean dataset ( such as si284 ) to learn the underlying patterns in the data , compared with a network trained on the smaller ( si284 subset ) or noisier ( tr05 multi ) dataset . Therefore , if we can show that a good classifier ( trained on si284 ) forces the feature extractor to learn useful patterns even from a small ( si284 subset ) or noisy dataset ( CHiME-4 ) , then this is evidence to support the good classifier hypothesis . Experimental setup . We use Kaldi ( Povey et al. , 2011 ) to extract 40-dimension filterbank features with three pitch features , and the toolkit ESPnet ( Watanabe et al. , 2018 ) to implement the end-toend models . We build bidirectional long short-term memory ( BLSTM ) ( Hochreiter & Schmidhuber , 1997 ) connectionist temporal classification ( CTC ) models ( Graves et al. , 2006 ) . All models have the same architecture . The network has 4 BLSTM layers . On top of each BLSTM layer is a projection layer , which is a linear layer with tanh activation functions . The top-most layer is a softmax layer . Thus , the model has 9 layers in total . Except for the softmax layer , each layer has 320 hidden units . The outputs are 26 characters , apostrophe , period , dash , space , noise , sos/eos tokens and some other special tokens , making 50 output labels in total . We used the adadelta ( Zeiler , 2012 ) optimization algorithm . We used WSJ dev93 and CHiME-4 dt05 multi isolated 1ch track as validation sets for the models trained on WSJ and CHiME-4 respectively . The training stops if after 5 epochs there is no improvement upon the best validation loss . Table 1 summarizes the experiment results in terms of character error rates ( CER ) , and indicates that the feature extractor can be trained to fit a frozen classifier . Furthermore , it shows that the classifier trained on the full si284 dataset is a good classifier – with the frozen classifier trained on full si284 , the model trained on the si284 subset and CHiME-4 has significiant lower CERs . Lower CERs also indicate that the network learns more about the underlying pattern . Thus , the experiments of training with the good classifier show that to capture underlying patterns , the weights of the feature extractor should have large changes . However , the vanishing gradient makes this almost impossible when all layers are trained jointly . These experiments support the good classifier hypothesis . When the classifier trained on the si284 subset is frozen , the feature extractor become well-trained compared with the under-trained feature extractor in joint training . When trained with a frozen classifier from the joint training , the weights of the feature extractor have larger changes , indicating that it fits the classifier . For the CHiME-4 experiments , however , fixing the classifier from the joint training only makes the top layer of the feature extractor have a larger weight change . We suggest that the reason for this is that when all layers are trained on the noisy dataset jointly , the middle layers overfit the bottom-most layers much faster than the bottom-most layers are able to learn input features . Thus , at the end of training , the middle layers are also over-trained and should be viewed as part of the over-trained classifier . Only the bottom-most layers should be viewed as the under-trained feature extractor . One piece of evidence for this is if we compare joint training on tr05 multi and on si284 , the changes of weights for the first BLSTM layer is comparable . However , for all other higher layers , the weights have larger changes if they are trained on tr05 multi , the noisy dataset . Our further experiments support this argument ( section 5 ) . In these experiments , training with the frozen good classifier terminates after more epochs compared to joint training or the training with poorer classifiers . This is consistent with the assumption that when training with a frozen good classifier , the main drawback of potentially small partial derivatives is slowing down training . Additionally , when taking the classifier from the model trained on full si284 , we only take the top-most BLSTM layer and the layers above it . We do not make the classifier deeper , since if we take and freeze more layers , the model will become very similar to the model trained on full si284 . As a result , it would be questionable whether the gains in accuracy are from better training through the fixed good classifier or because the model is indistinguishable from the model trained on full si284 . Finally , note that these experiments are conducted to support the good classifier hypothesis . The intention is not to perform transfer learning , although it is possible to develop a transfer learning method based on this approach .

Summary: This paper studies the common experimental finding that low level features trained end-to-end in a deep model converge (get "locked in place") earlier than higher level features, which may result in problematic undertraining. The focus of the study is not on skip connections, but really on getting adequate training in deeper networks. They posit a "good classifier hypothesis" where, once a deep network converged, they fix the top layers (the "good classifier") and train only the lower ones. They propose a "top-down training strategy" to search where to make the cut for the "top layers" of the "good classifier", based on the validation set.