Introduction
State-of-the-art speech recognition accuracy has significantly improved over the past few years since the application of deep neural networks BIBREF0 , BIBREF1 . Recently, it has been shown that with the application of both neural network acoustic model and language model, an automatic speech recognizer can approach human-level accuracy on the Switchboard conversational speech recognition benchmark using around 2,000 hours of transcribed data BIBREF2 . While progress is mainly driven by well engineered neural network architectures and a large amount of training data, the hidden Markov model (HMM) that has been the backbone for speech recognition for decades is still playing a central role. Though tremendously successful for the problem of speech recognition, the HMM-based pipeline factorizes the whole system into several components, and building these components separately may be less computationally efficient when developing a large-scale system from thousands to hundred of thousands of examples BIBREF3 .
Recently, along with hybrid HMM/NN frameworks for speech recognition, there has been increasing interest in end-to-end training approaches. The key idea is to directly map the input acoustic frames to output characters or words without the intermediate alignment to context-dependent phones used by HMMs. In particular, three architectures have been proposed for the goal of end-to-end learning: connectionist temporal classification (CTC) BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , sequence-to-sequence with attention model BIBREF8 , BIBREF9 , BIBREF10 , and neural network segmental conditional random field (SCRF) BIBREF11 , BIBREF12 . These end-to-end models simplify the pipeline of speech recognition significantly. They do not require intermediate alignment or segmentation like HMMs, instead, the alignment or segmentation is marginalized out during training for CTC and SCRF or inferred by the attention mechanism. In terms of the recognition accuracy, however, the end-to-end models usually lag behind their HMM-based counterparts. Though CTC has been shown to outperform HMM systems BIBREF13 , the improvement is based on the use of context-dependent phone targets and a very large amount of training data. Therefore, it has almost the same system complexity as HMM acoustic models. When the training data is less abundant, it has been shown that the accuracy of CTC systems degrades significantly BIBREF14 .
However, end-to-end models have the flexibility to be combined to mitigate their individual weaknesses. For instance, multitask learning with attention models has been investigated for machine translation BIBREF15 , and Mandarin speech recognition using joint Character-Pinyin training BIBREF16 . In BIBREF17 , Kim et al. proposed a multitask learning approach to train a joint attention model and a CTC model using a shared encoder. They showed that the CTC auxiliary task can help the attention model to overcome the misalignment problem in the initial few epochs, and speed up the convergence of the attention model. Another nice property of the multitask learning approach is that the joint model can still be trained end-to-end. Inspired by this work, we study end-to-end training of a joint CTC and SCRF model using an interpolated loss function. The key difference of our study from BIBREF17 is that the two loss functions of the CTC and attention models are locally normalized for each output token, and they are both trained using the cross entropy criterion. However, the SCRF loss function is normalized at the sequence-level, which is similar to the sequence discriminative training objective function for HMMs. From this perspective, the interpolation of CTC and SCRF loss functions is analogous to the sequence discriminative training of HMMs with CE regularization to overcome overfitting, where a sequence-level loss is also interpolated with a frame-level loss, e.g., BIBREF18 . Similar to the observations in BIBREF17 , we demonstrate that the joint training approach improves the recognition accuracies of both CTC and SCRF acoustic models. Further, we also show that CTC can be used to pretrain the neural network feature extractor to speed up the convergence of the joint model. Experiments were performed on the TIMIT database.
Segmental Conditional Random Fields
SCRF is a variant of the linear-chain CRF model where each output token corresponds to a segment of input tokens instead of a single input instance. In the context of speech recognition, given a sequence of input vectors of $T$ frames ${X} = ( {x}_1, \cdots , {x}_T )$ and its corresponding sequence of output labels ${y} = ( y_1, \cdots , y_J)$ , the zero-order linear-chain CRF defines the sequence-level conditional probability as P(y X) = 1Z(X) t=1T f ( yt, xt ), where $Z({X})$ denotes the normalization term, and $T=J$ . Extension to higher order models is straightforward, but it is usually computationally much more expensive. The model defined in Eq. ( "Segmental Conditional Random Fields" ) requires the length of ${X}$ and ${y}$ to be equal, which makes it inappropriate for speech recognition because the lengths of the input and output sequences are not equal. For the case where $T\ge J$ as in speech recognition, SCRF defines the sequence-level conditional probability with the auxiliary segment labels ${E} = ({e}_1, \cdots , {e}_J) $ as P(y, E X) = 1Z(X) j=1J f ( yj, ej, xj ), where $\mathbf {e}_j = \langle s_{j}, n_{j} \rangle $ is a tuple of the beginning ( ${X} = ( {x}_1, \cdots , {x}_T )$0 ) and the end ( ${X} = ( {x}_1, \cdots , {x}_T )$1 ) time tag for the segment of ${X} = ( {x}_1, \cdots , {x}_T )$2 , and ${X} = ( {x}_1, \cdots , {x}_T )$3 while ${X} = ( {x}_1, \cdots , {x}_T )$4 ; ${X} = ( {x}_1, \cdots , {x}_T )$5 and ${X} = ( {x}_1, \cdots , {x}_T )$6 denotes the vocabulary set; ${X} = ( {x}_1, \cdots , {x}_T )$7 is the embedding vector of the segment corresponding to the token ${X} = ( {x}_1, \cdots , {x}_T )$8 . In this case, ${X} = ( {x}_1, \cdots , {x}_T )$9 sums over all the possible ${y} = ( y_1, \cdots , y_J)$0 pairs, i.e.,
$$Z({X}) = \sum _{y,E} \prod _{j=1}^J \exp f \left( y_j, {e}_j, \bar{x}_j \right).$$   (Eq. 1)
Similar to other CRFs, the function $f(\cdot )$ is defined as
$$f \left( y_j, {e}_j, \bar{x}_t \right) = \mathbf {w}^\top \Phi (y_j, {e}_j, \bar{x}_j),$$   (Eq. 2)
where $\Phi (\cdot )$ denotes the feature function, and $\mathbf {w}$ is the weight vector. Most of conventional approaches for SCRF-based acoustic models use a manually defined feature function $\Phi (\cdot )$ , where the features and segment boundary information are provided by an auxiliary system BIBREF19 , BIBREF20 . In BIBREF21 , BIBREF12 , we proposed an end-to-end training approach for SCRFs, where $\Phi (\cdot )$ was defined with neural networks, and the segmental level features were learned by RNNs. The model was referred to as the segmental RNN (SRNN), and it will be used as the implementation of the SCRF acoustic model for multitask learning in this study.
Feature Function and Acoustic Embedding
SRNN uses an RNN to learn segmental level acoustic embeddings. Given the input sequence ${X} = ({x}_1, \cdots , {x}_T)$ , and we need to compute the embedding vector $\bar{x}_j$ in Eq. ( 2 ) corresponding to the segment ${e}_j = \langle s_j, n_j\rangle $ . Since the segment boundaries are known, it is straightforward to employ an RNN to map the segment into a vector as [ l hsj
hsj+1
$\vdots $
hnj ] = [ l RNN(h0, xsj)
RNN(hsj, xsj+1)
$\vdots $
RNN(hnj-1, xnj) ] where ${h}_0$ denotes the initial hidden state, which is initialized to be zero. RNN( $\cdot $ ) denotes the nonlinear recurrence operation used in an RNN, which takes the previous hidden state and the feature vector at the current timestep as inputs, and produce an updated hidden state vector. Given the recurrent hidden states, the embedding vector can be simply defined as $\bar{x}_j= {h}_{n_j}$ as in our previous work BIBREF12 . However, the drawback of this implementation is the large memory cost, as we need to store the array of hidden states $({h}_{s_j}, \cdots , {h}_{n_j})$ for all the possible segments $\langle s_j, n_j\rangle $ . If we denote $H$ as the dimension of an RNN hidden state, the memory cost will be on the order of $O(T^2H)$ , where $T$ is the length of $X$ . It is especially problematic for the joint model as the CTC model requires additional memory space. In this work, we adopt another approach that requires much less memory. In this approach, we use an RNN to read the whole input sequence as [ c h1
h2
$\vdots $
hT ] = [ l RNN(h0, x1)
RNN(h1, x2)
$\vdots $
RNN(hT-1, xT) ] and we define the embedding vector for segment ${e} = \langle k, t\rangle $ as xj = [ c hsj
hnj ] In this case, we only provide the context information for the feature function $\Phi (\cdot )$ to extract segmental features. We refer this approach as context-aware embedding. Since we only need to read the input sequence once, the memory requirement is on the order of $O(TH)$ , which is much smaller. The cost, however, is the slightly degradation of the recognition accuracy. This model is illustrated by Figure 1 .
The feature function $\Phi (\cdot )$ also requires a vector representation of the label $y_j$ . This embedding vector can be obtained using a linear embedding matrix, following common practice for RNN language models. More specifically, $y_j$ is first represented as a one-hot vector ${v}_j$ , and it is then mapped into a continuous space by a linear embedding matrix ${M}$ as
$${u}_j = {M v}_j$$   (Eq. 4)
Given the acoustic embedding $\bar{x}_j$ and label embedding $u_j$ , the feature function $\Phi (\cdot )$ can be represented as (yj, ej, xj) = (W1uj + W2xj + b), where $\sigma $ denotes a non-linear activation function (e.g., sigmoid or tanh); $W_1, W_2$ and $b$ are weight matrices and a bias vector. Eq. ( "Connectionist Temporal Classification " ) corresponds to one layer of non-linear transformation. In fact, it is straightforward to stack multiple nonlinear layers in this feature function.
Loss Function
For speech recognition, the segmentation labels ${E}$ are usually unknown in the training set. In this case, we cannot train the model directly by maximizing the conditional probability in Eq. ( "Segmental Conditional Random Fields" ). However, the problem can be addressed by marginalizing out the segmentation variable as Lscrf = - P(y X)
= - E P(y, E X)
= - E j f ( yj, ej, xj ) Z(X, y) + Z(X), where $Z({X}, {y})$ denotes the summation over all the possible segmentations when only ${y}$ is observed. To simplify notation, the objective function $\mathcal {L}_{\mathit {scrf}}$ is defined here with only one training utterance.
However, the number of possible segmentations is exponential in the length of ${X}$ , which makes the naïve computation of both $Z({X}, {y})$ and $Z({X})$ impractical. To address this problem, a dynamic programming algorithm can be applied, which can reduce the computational complexity to $O(T^2\cdot |\mathcal {Y}|)$ BIBREF22 . The computational cost can be further reduced by limiting the maximum length of all the possible segments. The reader is referred to BIBREF12 for further details including the decoding algorithm.
Connectionist Temporal Classification 
CTC also directly computes the conditional probability $P(y \mid X)$ , with the key difference from SCRF in that it normalizes the probabilistic distribution at the frame level. To address the problem of length mismatch between the input and output sequences, CTC allows repetitions of output labels and introduces a special blank token ( $-$ ), which represents the probability of not emitting any label at a particular time step. The conditional probability is then obtained by summing over all the probabilities of all the paths that corresponding to $y$ after merging the repeated labels and removing the blank tokens, i.e., P(y X) = (y) P(X), where $\Psi (y)$ denotes the set of all possible paths that correspond to $y$ after repetitions of labels and insertions of the blank token. Now the length of $\pi $ is the same as $X$ , the probability $P(\pi \mid X)$ is then approximated by the independence assumption as P(X) t=1T P(t xt), where $\pi _t $ ranges over $\mathcal {Y}\cup \lbrace -\rbrace $ , and $-$0 can be computed using the softmax function. The training criterion for CTC is to maximize the conditional probability of the ground truth labels, which is equivalent to minimizing the negative log likelihood: Lctc = -P(y X), which can be reformulated as the CE criterion. More details regarding the computation of the loss and the backpropagation algorithm to train CTC models can be found in BIBREF23 .
Joint Training Loss
Training the two models jointly is trivial. We can simply interpolate the CTC and SCRF loss functions as L = Lctc + (1-)Lscrf, where $\lambda \in [0, 1]$ is the interpolation weight. The two models share the same neural network for feature extraction. In this work, we focus on the RNN with long short-term memory (LSTM) BIBREF24 units for feature extraction. Other types of neural architecture, e.g., convolutional neural network (CNN) or combinations of CNN and RNN, may be considered in future work.
Experiments
Our experiments were performed on the TIMIT database, and both the SRNN and CTC models were implemented using the DyNet toolkit BIBREF25 . We followed the standard protocol of the TIMIT dataset, and our experiments were based on the Kaldi recipe BIBREF26 . We used the core test set as our evaluation set, which has 192 utterances. Our models were trained with 48 phonemes, and their predictions were converted to 39 phonemes before scoring. The dimension of $\mathbf {u}_j$ was fixed to be 64, and the dimension of $\mathbf {w}$ in Eq. ( 2 ) is also 64. We set the initial SGD learning rate to be 0.1, and we exponentially decay the learning rate by 0.75 when the validation error stopped decreasing. We also subsampled the acoustic sequence by a factor of 4 using the hierarchical RNN as in BIBREF12 . Our models were trained with dropout regularization BIBREF27 , using a specific implementation for recurrent networks BIBREF28 . The dropout rate was 0.2 unless specified otherwise. Our models were randomly initialized with the same random seed.
Baseline Results
Table 1 shows the baseline results of SRNN and CTC models using two different kinds of features. The FBANK features are 120-dimensional with delta and delta-delta coefficients, and the fMLLR features are 40-dimensional, which were obtained from a Kaldi baseline system. We used a 3-layer bidirectional LSTMs for feature extraction, and we used the greedy best path decoding algorithm for both models. Our SRNN and CTC achieved comparable phone error rate (PER) for both kinds of features. However, for the CTC system, Graves et al. BIBREF29 obtained a better result, using about the same size of neural network (3 hidden layers with 250 hidden units of bidirectional LSTMs), compared to ours (18.6% vs. 19.9%). Apart from the implementation difference of using different code bases, Graves et al. BIBREF29 applied the prefix decoding with beam search, which may have lower search error than our best path decoding algorithm.
Multitask Learning Results
Table 2 shows results of multitask learning for CTC and SRNN using the interpolated loss in Eq. ( "Joint Training Loss" ). We only show results of using LSTMs with 250 dimensional hidden states. The interpolation weight was set to be 0.5. In our experiments, tuning the interpolation weight did not further improve the recognition accuracy. From Table 2 , we can see that multitask learning improves recognition accuracies of both SRNN and CTC acoustic models, which may due to the regularization effect of the joint training loss. The improvement for FBANK features is much larger than fMLLR features. In particular, with multitask learning, the recognition accuracy of our CTC system with best path decoding is comparable to the results obtained by Graves et al. BIBREF29 with beam search decoding.
One of the major drawbacks of SCRF models is their high computational cost. In our experiments, the CTC model is around 3–4 times faster than the SRNN model that uses the same RNN encoder. The joint model by multitask learning is slightly more expensive than the stand-alone SRNN model. To cut down the computational cost, we investigated if CTC can be used to pretrain the RNN encoder to speed up the training of the joint model. This is analogous to sequence training of HMM acoustic models, where the network is usually pretrained by the frame-level CE criterion. Figure 2 shows the convergence curves of the joint model with and without CTC pretraining, and we see pretraining indeed improves the convergence speed of the joint model.
Conclusion
We investigated multitask learning with CTC and SCRF for speech recognition in this paper. Using an RNN encoder for feature extraction, both CTC and SCRF can be trained end-to-end, and the two models can be trained together by interpolating the two loss functions. From experiments on the TIMIT dataset, the multitask learning approach improved the recognition accuracies of both CTC and SCRF acoustic models. We also showed that CTC can be used to pretrain the RNN encoder, speeding up the training of the joint model. In the future, we will study the multitask learning approach for larger-scale speech recognition tasks, where the CTC pretraining approach may be more helpful to overcome the problem of high computational cost.
Acknowledgements
We thank the NVIDIA Corporation for the donation of a Titan X GPU.

Question: Can SCRF be used to pretrain the model?
Answer: No