# Efficient DNN Model for Word Lip-Reading
## 2 Related Research  
There are many studies on word lip-reading techniques. Here, this paper focuses on research targeting LRW. LRW is a dataset published by [PERSON] et al. in 2016 [4] ([[https://www.robots.ox.ac.uk/~vgg/data/lip_reading/lrv1.html](https://www.robots.ox.ac.uk/~vgg/data/lip_reading/lrv1.html)]([https://www.robots.ox.ac.uk/~vgg/data/lip_reading/lrv1.html](https://www.robots.ox.ac.uk/~vgg/data/lip_reading/lrv1.html)), accessed on 26 April 2023) and is a large-scale lip-reading research dataset containing 500 English words. LRW is used as a benchmark in the lip-reading field. LRW contains utterance scenes clipped from news and discussion programs broadcast from 2010 to 2016 by the British Broadcasting Corporation (BBC), which collectively manages radio and television in the United Kingdom. While most datasets are utterance scenes in which the speaker was recording in a roughly specified posture, LRW has the feature of recording utterance scenes in a natural posture even though it is a TV program. The number of speakers is more than 1000. LRW consists of three types of data: training data, validation data, and test data, and provides video data containing 488,766 scenes, 25,000 scenes, and 25,000 scenes, respectively. The train data contain 800-1000 scenes per word, and the validation and test data contain 50 scenes per word. All scenes are extracted face images with 256 \(\times\) 256 pixels, the frame rate is 25 fps, the scene length is 1.16 s, and the number of frames is 29. The total recording time of the three types is about 173 h.  
CAS-VSR-W1k [16], which contains 1000 Chinese words, is a large-scale open dataset containing word utterance scenes. The state-of-the-art (SOTA) recognition accuracies of LRW and CAS-VSR-W1k are 94.1% [18] and 55.7% [19], respectively. Academically, it is desirable to target CAS-VSR-W1k, which has a high degree of task difficulty. Since we could not obtain CAS-VSR-W1k, we target LRW. [PERSON] et al., who published LRW, proposed four convolutional neural network (CNN) models based on VGG-M [4]. Among the four models, the multiple towers model obtained the highest recognition rate of 66.1%. This model has a structure in which the convolutional layers for all frames are provided in the first layer. The outputs of all subsequent convolutional layers are connected to one and input to the second convolutional layer. The following year, [PERSON] et al. proposed a new network with a watch, listen, attend, and spell (WLAS) structure, obtaining 76.2% [10]. In WLAS, the encoder consists of a CNN model that is an improved version of VGG-M, which extracts features from each input frame image, and a long short-term memory (LSTM) that summarizes the output of the features from the CNN model. The decoder consists of LSTM, attention, and softmax. According to the paper with the code site ([[https://paperswithcode.com/sota/lipreading-on-lip-reading-in-the-wild](https://paperswithcode.com/sota/lipreading-on-lip-reading-in-the-wild)]([https://paperswithcode.com/sota/lipreading-on-lip-reading-in-the-wild](https://paperswithcode.com/sota/lipreading-on-lip-reading-in-the-wild)), accessed on 26 April 2023), SOTA in LRW is currently the result of [PERSON] et al. [18]. The model architecture consists of 3D-Conv + ResNet18 in the front stage and a Temporal model in the backstage with a mouth region of interest (ROI) as input. The temporal model is an ensemble of three different models; densely connected temporal convolutional networks (DC-TCNs), multi-scale temporal convolutional networks (MS-TCNs), and bidirectional gated recurrent units (BGRUs). In addition to the model architecture, they applied data augmentation, self-distillation, and word boundary indicators to improve the recognition accuracy. Many other papers have recently discussed the model training strategy, but the model architecture is often 3D-Conv + ResNet18 + MS-TCN [6; 8; 20].