# Chinese Lip-Reading Research Based on ShuffleNet and CBAM
## 1 Introduction  
Lip reading is the task of recognizing contents in a video only based on visual information. It is the intersection of computer vision and natural language processing. At the same time, it can be applied in a wide range of scenarios such as human-computer interaction, public security, and speech recognition. Depending on the mode of recognition, lip reading can be divided into audio-visual speech recognition (AVSR) and visual speech recognition (VSR). AVSR refers to the use of image processing capabilities in lip reading to aid speech recognition systems. In VSR, speech is transcribed using only visual information to interpret tongue and teeth movements. Depending on the object of recognition [1], lip reading can be divided into isolated lip recognition methods and continuous lip recognition methods. Isolated lip recognition method targets numbers, letters, words, or phrases, which can be classified into limited categories visually. The continuous lip recognition method targets phonemes, visemes (the basic unit of visual information) [2], and visually indistinguishable characters.  
In 2016, Google [3] and the University of Oxford designed and implemented the first sentence-level lip recognition model, named LipNet. [PERSON], [PERSON] et al. [4] used the lip recognition method of CNN and LSTM to solve the complex speech recognition problem that the HMM network could not solve. In 2019, as the attention mechanism was introduced into the field of lip recognition, [PERSON] et al. [5] proposed a lip-reading recognition system using the CNN-Bi-GRU-Attention fusion neural network model, and the final recognition accuracy reached 86.8%. In 2021, [PERSON]. [6] improved the above fusion lip recognition model and proposed the HLR-Net model. The model mainly was composed of the fusion model of Inception-Bi-GRU-Attention and used the CTC loss function to match the input and output, and its recognition accuracy reached 92%.  
However, the research mentioned above involves English datasets, and the development of Chinese lip-reading technology is still in the initial stage. Compared with English which consists of only letters, Chinese is more complex. This is because Chinese Pinyinhas more than 1000 pronunciation combinations and the number of Chinese characters is more than 9000. Moreover, the deficiency of Chinese datasets also makes lip reading more challenging.  
Additionally, in recent years, building deeper and larger neural networks is a primary trend in the development of major visual tasks [7; 8; 9], which requires computation at billions of FLOPs. The high cost limits the practical deployment of lip-reading models.  
In this paper, we aim to propose a deep-learning model designed for on-device application on our self-built dataset called 'Databox'. Our model is improved based on the current state-of-the-art methodology consisting of a ResNet network and a Temporal Convolutional Network. We replace ResNet with a lightweight convolutional network called ShuffleNet with a plug-in attention module called Convolutional Block Attention. In Section 2, we give a detailed description of all the parts of the model. In Section 3, we present and analyze the results of the experiments. In Section 4, we conclude that our model architecture not only reduces computation but also maintains comparable accuracy that is suitable for mobile platforms such as drones, robots, and phones with limited computing power.