# Efficient End-to-End Sentence-Level Lipreading with Temporal Convolutional Networks
## 3 Proposed Architecture
### Connectionist Temporal Classification (CTC)  
The CTC objective function [17] was originally widely used in speech recognition. In view of the similarity between speech recognition and lipreading, CTC was introduced in lipreading.  
Figure 4: The basic architecture of the Temporal Convolutional Network (TCN). (1) A sequence of feature maps (\(x_{0},x_{1},\ldots,x_{T-1},x_{T}\)) as input. (2) The feature maps are learned and decoded by Temporal Convolutional Network. (3) The TCN architecture finally produces a sequence of learning results (\(y_{0},y_{1},\ldots,y_{T-1},y_{T}\)).  
The core step of CTC is to convert the output of each time step of the sequence model into a probability distribution in the label sequences. The softmax activation function of the CTC network converts the output into a probability. The number of units is one more unit than the number of labels \(L\); therefore, the output of each softmax layer can denote the probability distribution of the corresponding label.  
Suppose a given input sequence \(\mathbf{x}\) of length \(T\), followed by a Bi-LSTM recurrent neural network layer with \(m\) input, \(n\) output, and \(w\) weight. Therefore, define a continuous mapping to denote Bi-LSTM, \(\mathbb{N}_{w}:(\mathbb{R}^{m})^{T}\mapsto(\mathbb{R}^{n})^{T}\). Then, \(\mathbf{y}=\mathbb{N}_{w}(\mathbf{x})\) becomes the output of the sequence model(e.g., Bi-LSTM) and defines \(y_{k}^{t}\) as the probability distribution of output \(k\) in time step \(t\). We describe an alphabet \(L^{\ast}\)=\(L\cup\{blank\}\). For each true label path, we obtain a probability by Formula (3):  
\[p(\pi|\mathbf{x})=\prod_{t=1}^{T}y_{\pi_{t}}^{t},\forall\pi\in L^{\cdot T} \tag{3}\]  
Formula (3) is the product of the probability distribution under a \(\pi\) path. There are many such paths from output to label. Given a mapping function from output to label \(\beta:L^{\cdot T}\mapsto L^{\preceq T}\), where \(L^{\preceq T}\) is the combination of a series of possible labels. In the next step, we delete blank and duplicate labels in all paths (e.g., \(\beta(c-cd-)=\beta(-cc-ccdd-)=ccd\)). For any given label \(l\in L^{\preceq T}\), through the inverse mapping of \(\beta^{-1}(l)\), we can obtain all of its \(\pi\) paths. Calculate the probability sum of all \(\pi\) paths by Formula (4):  
\[p(l|\mathbf{x})=\sum_{\pi\in\beta^{-1}(l)}p(\pi|\mathbf{x}) \tag{4}\]  
Through the above Formula (4), we only need to obtain the most probable labeling of the input sequence as the output of CTC, as shown in Formula (5):  
\[h(\mathbf{x})=arg\max_{l\in L^{\preceq T}}p(l|\mathbf{x}) \tag{5}\]  
Finally, we use the CTC network to minimize Formula (6) as the training goal, and constantly update the weight parameters of the entire model:  
\[\zeta_{ctc}=-\ln h(\mathbf{x}) \tag{6}\]  
In terms of lipreading, our dataset is video data and its corresponding text. Unfortunately, it is difficult to align video data and text in units. If we directly train the model without using alignment, it will be difficult for the model to converge due to the difference in people's speech speed or the distance between characters. From the above description of CTC, we know that CTC is a solution that avoids manual alignment of input and output and is very suitable for lipreading or speech recognition applications. Therefore, CTC is a sensible choice for the lipreading task.