Backpropagation in Sequential Deep Neural Networks

Most previous work applying neural networks to problems in speech processing has combined the output of a static network trained over a sliding window of input with an HMM or CRF to model linear-chain dependencies in the output. The recently developed Sequential Deep Neural Network (SDNN) model allows sequential dependencies between internal hidden units, allowing them to potentially detect long-range phenomena. During the upward propagation phase, a binary forward-backward (Baum-Welch) computation is performed for each hidden unit, passing information along the extent of the chain. Then the gradient of the SDNN error can be computed exactly via a modified backpropagation algorithm that also includes an information-passing Baum-Welch-like step. The model introduces minimal computational overhead compared to other DNN approaches to sequential labeling, and achieves comparable performance with a much smaller model (in terms of number of parameters). Experiments on the TIMIT phone recognition dataset show that using sequential information at all layers improves accuracy over baseline models that use a sequence model only in the output. 1 Learning deep representations for sequential labeling Sequential labeling problems occur in a variety of settings, including speech recognition, audio and video processing, natural language processing, and bioinformatics. The goal of sequential labeling is to map whole sequences of inputs over observations {x1, x2, . , xT } to sequences of labels {y1, y2, . , yT }, where the length of the sequence T may vary from instance to instance, and where dependencies between observations or labels at different positions within a sequence are used to produce a better labeling than might be achieved by labeling each yi independently. In Conditional Random Fields (CRFs) [Lafferty et al., 2001], a Markov Random Field (MRF) is defined over the label sequence whose parameters depend on the input. Typically, the graphical model structure is a linear chain so that globally conditioned on the input, each label is conditionally independent of the other labels given its neighboring labels. CRFs allow features of the input to be defined at arbitrary distance from the associated label, but the user must consciously design such features to allow long-distance dependencies and, moreover, the feature functions may only be fixed length. Feature design is a difficult, task-specific problem, and it is especially difficult to hand-design effective long-range features for tasks such as speech recognition, where the input is a relatively low-level representation of the acoustic signal. On the other hand, results in speech science suggest that longer-range features (that is, longer than the typical 25ms frame width) may be useful for speech perception, particularly in noisy environments [Hermansky and Sharma, 2002]. Several methods have been proposed to introduce hidden variables to CRFs that might be capable of modeling regularities in the data that are not explicit in the features but nevertheless aid in classification. We focus here on approaches that were applied to phone recognition, which is a prototypical sequential labeling task, and the subject of our experiments. The hidden CRF (HCRF) appends a