# Efficient End-to-End Sentence-Level Lipreading with Temporal Convolutional Networks
## 1 Introduction  
Lipreading, also known as visual language recognition, refers to decoding the content of the spoken text based on the visual information of the speaker's lip movement. It has a wide range of applications values in speech recognition [1], public safety [2], intelligent human-computer interaction [3], visual synthesis, etc. Traditionally, the lipreading method can follow two stages. First, it extracted features from the mouth region. Discrete Cosine Transform [4, 5] was considered the most popular feature extractor, and then it was fed to the Hidden Markov Model (HMM) [1, 6, 7]. At the same time, there are some similar methods proposed: the difference is that the feature extractor is replaced with a deep autoencoder, and HMMS was replaced with Long-Short Term Memory (LSTM) [8, 9]. Deep learning methods have achieved great success [10, 11] in many complex tasks based on traditional machine learning [12, 13, 14]. Convolutional Neural Networks (CNNs) show superior performance in image and video feature extraction compared to traditional methods. For example, [PERSON] et al. [15] present a deep learning architecture for lipreading and audiovisual word recognition. [PERSON] et al. [16] proposed an end-to-end visual speech recognition system based on fully connected layers and LSTM networks. At this stage, there are two solutions for the lipreading architecture based on deep learning, which are divided into Connectionist Temporal Classification(CTC) [17]-based speech recognition [18, 19] technology, and the attention-based sequence-to-sequence (seq2 seq) [20] neural network translation model. Speech recognition [18, 21] technology-based on the CTC approach has made a great breakthrough. For the lipreading problem, [PERSON]. [22, 23, 24] used Convolutional NeuralNetwork (CNN) [25; 26] as a feature extractor, Recurrent Neural Network(RNN) [27; 28] as the feature learner, and CTC [17] as the objective function, training an end-to-end sentence-level lipreading architecture. Its architecture outperforms experienced human lipreaders on the GRID dataset [29]. However, these architectures have two main problems. On the one hand, a simple feature extractor is not competent for feature extraction of video data. On the other hand, the use of RNN will have the defects of vanishing or exploding gradients. The attention-based sequence-to-sequence model was first used in the neural network translation model [20] to solve the problem that the input sequence and output sequence are not aligned in time. For the lipreading problem, [PERSON]. [30; 31] use the Attention-based seq2 seq model to build a WAS (Watch, Attend and Spell) architecture. Outstanding performance in LRW [32] and GRID [29] datasets shows a Word Error Rate(WER) of 23.8% on the LRW [32] dataset. However, recent work [33; 34] shows that the attention-based sequence-to-sequence model cannot correctly align with the output sequence for longer input sequences, so it is hard to converge during the entire training process. However, recent results [35] show that the convolutional architecture performs better than recurrent networks on audio synthesis and machine translation tasks. [PERSON]. [36] proposed a general TCN model and did a series of evaluation experiments for all serialization tasks. The results show that the TCN performs better than the primary recurrent neural network(e.g., BLSTM, BGRU) in a broad range of sequence model tasks. In this work, we propose a state-of-the-art model that improves the performance of lipreading and the speed of convergence. First, we discard the basic BGRU or BLSTM neural network layer and replace it with a TCN [36]. Secondly, we propose a more efficient and complex feature extractor based on a 3D convolutional network and ResNet50 [37]. Its efficiency is greatly improved compared with standard feature extractors. Finally, we use the CTC [17] objective function as a decoder to implement an end-to-end sentence-level lipreading architecture.