Introduction
Very deep CNNs achieve state-of-the-art results on various tasks BIBREF1 in computer vision. Network depth has been crucial in obtaining those leading results BIBREF1, BIBREF2. Naïve deep stacking of layers typically leads to a vanishing/exploding gradients problem, making convergence difficult or impossible. For example, VGGNet BIBREF1 only uses 18 layers. Normalization methods, including batch normalization BIBREF3, layer normalization BIBREF4 and weight normalization BIBREF5, allow deeper neural nets to be trained. Unfortunately, these normalization methods make training stability sensitive to other factors, such as SGD, dropout, and the estimation of normalization parameters. Accuracy often saturates and degrades as network depth increases BIBREF6, BIBREF7.
ResNet BIBREF8 uses shortcut connections (SC) and batch normalization (BN), allowing the training of surprisingly deep architectures with dramatic accuracy improvements. Since its invention, ResNet has dominated the field of computer vision. The later state-of-the-art-model, DenseNet BIBREF9, also uses SC and BN. Besides success in computer vision, ResNet has also performed well in acoustic models for speech recognition BIBREF10, BIBREF11.
An alternative solution to the problem of vanishing/exploding gradients is self-normalizing neural networksBIBREF0. SNNs use the scaled exponential linear unit (SELU) activation function to induce self-normalization. SNNs have been shown to converge very deep networks without shortcut connections or batch normalization. SNNs are also robust to perturbations caused by training regularization techniques.
Very deep convolutional neural network acoustic models are computationally expensive when used for speech recognition. Several techniques have been explored to improve inference speed on commodity server CPUs. Batching and lazy evaluation have been shown to improve inference speed on CPUs BIBREF12 for neural networks of all types. Specifically for speech recognition, running inference at a decreased frame rate BIBREF13 has also been shown to reduce computation cost without affecting accuracy.
Related Work ::: Residual Learning
ResNet BIBREF8 solves many problems in training very deep CNNs. The key ResNet innovation is the shortcut connections shown in Figure FIGREF1. Figure FIGREF1 is a typical building block of ResNet. The input to the block, $x$, will go through both the original mapping $F(x)$ (weight layers, RELU activations and batch normalization BIBREF3) and the identity shortcut connection. The output, $y$, will be $F(x)+x$. The authors of BIBREF8 hypothesize that the so-called residual mapping of $y=F(x)+x$ should be easier to optimize than the original mapping of $y=F(x)$. The design of the special building block is motivated by the observation in BIBREF6, BIBREF7 that accuracy degrades when more layers are stacked onto an already very deep CNN model. If the added layers can be constructed as identity mappings, the deeper model should not have worse training error than the original shallower model without these added layers. The degradation actually suggests that the optimizer has difficulties in approximating identity mappings. With the identity shortcut connections in the ResNet building block, the optimizer can simply drive the layer weights toward zero to make the block identity mapping. ResNet-style CNNs have maintained state-of-the-art results and have inspired other model structures BIBREF9, BIBREF14.
Related Work ::: Batch Normalization
Besides the shortcut connections shown in Figure FIGREF1, batch normalization (BN) BIBREF3 is also an important feature of ResNet. BN is designed to reduce internal covariate shift, defined as the change in the distribution of network activations due to the change in network parameters, during training. This ensures better and faster convergence of the training process. BN is achieved by whitening the input of each layer, but full whitening of each layer’s inputs is costly and not differentiable everywhere. Instead of whitening the features in layer inputs and outputs jointly, each scalar feature is normalized independently to zero mean and unit variance. For a layer with d-dimensional input $x = (x(1) . . . x(d))$, each dimension will be normalized as:
BN also ensures that the normalization can represent the identity transform by introducing a pair of parameters $\gamma ^{(k)}$, $\beta ^{(k)}$, which scale and shift the normalized value $\widehat{x}^{(k)}$:
In mini-batch based stochastic optimization, the mean $\mathbf {E}[x^{(k)}]$ and variance $\mathbf {Var}[x^{(k)}]$ are estimated within each mini-batch.
BN has been successfully adopted in various tasks, but training with BN can be perturbed by many factors such as SGD, dropout, and the estimation of normalization parameters. Moreover, in order to fully utilize BN, samples in each mini-batch must be i.i.d BIBREF15. However, state-of-the-art speech recognition requires sequence level training of the acoustic model BIBREF16. In sequence level training, a mini-batch consists of all the frames of a single utterance, and the frames are highly correlated to each other. This violates the i.i.d requirement of BN, making batch normalization very challenging to use with sequence training.
Related Work ::: Self-Normalizing Neural Networks
BIBREF0 introduces self-normalizing neural networks (SNNs) in which neuron activations automatically converge towards zero mean and unit variance. The key to inducing the self-normalizing properties in SNNs is the special activation function, the scaled exponential linear unit (SELU), formulated as:
with $\alpha \approx 1.6733$ and $\lambda \approx 1.0507$.
The magic numbers of $\alpha $ and $\lambda $ are obtained by solving fixed point equations to give the activation function the following characteristics, which ensures the self-normalizing property BIBREF0:
Negative and positive values for controlling the mean
Saturation regions (derivatives approaching zero) to dampen the variance if it is too large in the lower layer
A slope larger than one to increase the variance if it is too small in the lower layer
A continuous curve
The shape of SELU activation function is shown in Figure FIGREF8. Using SELU, SNNs push neuron activations to zero mean and unit variance. This gives us the same effect as batch normalization without being prone to the perturbations discussed in Section SECREF3.
Training Self-Normalizing very deep CNNs
We revise the model topology from BIBREF8 to build the proposed Self-Normalizing Deep CNNs (SNDCNN) for hybrid speech recognition system BIBREF17. The building block for SNDCNN is shown in Figure FIGREF13. Comparing to Figure FIGREF1, the shortcuts and batch normalization are removed, and the activation function is changed to SELU. In effect, we make a self-normalizing ResNet.
We verify the Self-Normalizing property by observing the trend of mean and variance in the SELU activation outputs during training. The model topology is a 50-layer CNN obtained by removing SC and BN from ResNet-50. We call this topology SNDCNN-50. Model parameters are initialized as instructed in BIBREF0. In Figures FIGREF14 and FIGREF15, we plot the mean and variance trend of the 1st, 23rd, 46th, 47th, and 49th layers of SNDCNN-50 and the 23rd layer of SNDCNN-24. The mean and variance are computed across frames within a mini-batch (256 frames). Each data point is obtained by averaging all the units in the same layer. The x-axis is training time, and we collect statistics from 33k mini-batches to draw each curve.
In the SNDCNN-50 case, we can see that the outputs of 1st and middle (23rd) layers follow the claims in BIBREF0 nicely, but the last several layers do not. We find that the non-self-normalizing phenomenon becomes significant only after the 46th layer. As shown in Figure FIGREF14 and FIGREF15, the 46th layer almost has mean $=$ 0 and variance $=$ 1, but the following layers are worse. We verify that the non-self-normalizing phenomenon is not caused by the depth of the neural network but by the distance to the output layer. The 23rd layer of SNDCNN-24 has the non-self-normalizing phenomenon, similar to the one seen in the 49th layer of SNDCNN-50, while the 23rd layer of SNDCNN-50 has a very nice self-normalizing property. We suspect that the back propagation path has to be long enough to effectively train the neural network's parameters to ensure the self-normalizing property. Although the last layers do not strictly follow BIBREF0's self-normalizing claim, the mean and variance are reasonable (mean $<$ 0.8, variance $<$ 9) even after 109 million mini-batches (28 billion training samples).
We also tried different kinds of initialization for the network. Our findings indicate that as long as training starts normally, the trend of the mean and variance will follow the patterns seen in Figures FIGREF14 and FIGREF15.
Removing SC and BN simplifies the model structure and speeds up both training and inference. Removing BN also solves the sequence level training problem discussed in Section SECREF3. More importantly, we always observe as good or better accuracy (WER) with the proposed simplified model structure.
Experiments
All the data used in this paper comes from Siri internal datasets (en_US and zh_CN). All the models are trained with Blockwise Model-Update Filtering (BMUF) BIBREF18 with 32 GPUs. A 4-gram language model is used in all experiments. 40 dimensional filter bank feature is extracted with 25ms window and 10ms step size. All the models use a context window of 41 frames (20-1-20) as the visible states BIBREF19.
Experiments ::: Accuracy
Table TABREF21 compares WER of different model topologies for en_US. The training data contains 300 hours of speech and the testing data has 7 hours of speech. From Table TABREF21, we have the following observations:
[Row 1-4 vs. Row 5-8] Deep CNN models show advantage in terms of WER against shallower DNNs
[Row 3 vs. Row 4] [Row 7 vs. Row 8] SELU activation makes the training of very deep models (with no SC&BN) feasible
[Row 1 vs. Row 2] [Row 5 vs. Row 6] SELU activation is no worse than RELU in DNN or ResNet topology.
[Row5 vs. Row 8] SNDCNN obtains better WER than ResNet
Table TABREF22 compares character error rate (CER) of different model topologies for zh_CN. The training data contains 4000 hours of speech and the testing data consists of 30 hours of speech. From Table TABREF22, we find that in order to make the training of very deep CNNs feasible, we must use at least one of the following three techniques: batch normalization, shortcut connection, and SELU activation. The WERs of different topologies with the same depth are actually very similar. This phenomenon suggests that depth could be the key to better accuracy. The proposed SNDCNN has slightly better WER than ResNet.
Table TABREF23 compares en_US WER of ResNet-50 and SNDCNN-50 with 10000 hours of training data and 7 hours of testing data. In this experiment, the proposed SNDCNN has much better WER than ResNet.
Experiments ::: Speedup
Table TABREF24 shows the relative computation speedups (frames per second) of the variants considered in Table TABREF22. From Table TABREF22, we know that the 4 models in Table TABREF24 have very similar WER. but from Table TABREF24, we can find that removal of BN and SC results in significant speedup in both training and inference. The speedup (especially in inference) is very important in deploying SNDCNN-50 in production systems where minimising latency is essential.
Inference Performance Optimization
We already achieve significant inference speedup by removing BN and SC from ResNet-50 as discussed in Section SECREF25. Further inference optimization for SNDCNN-50 was investigated, particularly frame-skipping and multi-threaded lazy computation.
Frame-skipping: our acoustic model targets tied HMM (hidden Markov model) states, running at 100 frames per second, but the predictions do not frequently change between frames. Human speech rarely has more than 10 phonemes per second. By simply skipping and duplicating two thirds of frames, we reduce the required computation by 3x which translates into 47.2% latency reduction as shown in Table TABREF27. Note that usually skipping frames will result in some WER degradation BIBREF13 and we indeed observed that in our experiments with shallower models (10 layer, 2 convolution layer plus 8 fully connected) even when we skip only half of the frames. However, with SNDCNN-50, we can skip up to two thirds of frames with no degradation on WER.
Multi-thread lazy computation: as shown in Figure FIGREF26, we split the acoustic model into two parts: front and back. We use two threads to do the inference independently. Thread 1 will do the inference of the front part which contains the input and hidden layers. Thread 2 will do the inference of the back part which contains the output layer. The outputs target tied HMM (hidden Markov model) states, and can easily be more than 10 thousand. As performing inference for the entire layer is expensive, we only compute the outputs that are needed by the decoding graph instead of computing every output of the layer. By doing this “lazy" on-demand inference, we save a lot of computation in the large output layer, which translates into a 10.8% latency reduction as shown in Table TABREF27.
Conclusions
In this paper, we proposed a very deep CNN based acoustic model topology SNDCNN, by removing the SC/BN and replacing the typical RELU activations with scaled exponential linear unit (SELU) in ResNet-50. This leverages self-normalizing neural networks, by use of scaled exponential linear unit (SELU) activations, to train very deep convolution networks, instead of residual learning BIBREF8). With the self-normalization ability of the proposed network, we find that the SC and BN are no longer needed. Experimental results in hybrid speech recognition tasks show that by removing the SC/BN and replacing the RELU activations with SELU in ResNet-50, we can achieve the same or lower WER and 60%-80% training and inference speedup. Additional optimizations in inference, specifically frame skipping and lazy computation with multi-threading, further speed up the SNDCNN-50 model by up to 58% which achieves production quality accuracy and latency.
Acknowledgments
The authors would like to thank Professor Steve Young, Bing Zhang, Roger Hsiao, Xiaoqiang Xiao , Chao Weng and Professor Sabato Marco Siniscalchi for valuable discussions and helps.

Question: What other model inference optimization schemes authors explore?
Answer:  shortcut connections, batch normalization (BN), self-normalizing neural networks (SNNs)

 frame-skipping, multi-threaded lazy computation

Question: On what dataset is model trained/tested?
Answer: Siri internal datasets (en_US and zh_CN)

Siri internal datasets (en_US and zh_CN)