# Chinese Lip-Reading Research Based on ShuffleNet and CBAM
## 3 Experiment  
### Dataset  
Because Chinese lip recognition technology is in its initial stage, the quality and number of lip-reading datasets available still have a long way to go. Most of the current lip-reading datasets are in English. There are a few influential ones such as:  
(1). The AVLetters dataset is the first audio-visual speech dataset which contains 10 speakers, each of whom makes three independent statements of 26 English letters. There are 780 utterances in total. (2). The XM2 VTS dataset includes 295 volunteers, each of whom reads two-digit sequences and phonetically balanced sentences (10 numbers, 7 words) at normal speaking speed. There are 7080 utterance instances in total. (3). The BANCA dataset is recorded in four different languages (English, French, Italian, Spanish) and filmed under three different conditions (controlled, degraded and adverse) with a total of 208 participants and nearly 30,000 utterances. (4). The OuluVS dataset aims to provide a unified standard for performance evaluation of audio-visual speech recognition systems. It contains 20 participants, each of whom states 10 daily greeting phrases 5 times for a total of 1000 utterance instances. (5). The LRW dataset is derived from BBC radio and television programs instead of being recorded by volunteers. It selects the 500 most frequent words and captures short video of the speakers saying these words, thus there are more than 1000 speakers and more than 550 million utterances. To some extent, the LRW dataset meets the requirements of deep learning in terms of data volume. Existing advanced methods trained on the LRW dataset are shown in Table 2. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Year** & **Method** & **Frontend** & **Backend** & **Input Size** & **LRW** \\ \hline
2016 & [PERSON] et al. [26] & VGGM & - & 112 \(\times\) 112 & 61.1\% \\
2017 & [PERSON] et al. [27] & ResNet-34 & BiLSTM & 112 \(\times\) 112 & 83.5\% \\
2019 & [PERSON] et al. [28] & Multi-Grained ResNet-18 & Conv BLSTM & 88 \(\times\) 88 & 83.3\% \\
2019 & [PERSON] et al. [29] & Two-Stream ResNet-18 & BLSTM & 112 \(\times\) 112 & 84.1\% \\
2020 & [PERSON] et al. [30] & ResNet-18 & BiGRU & 88 \(\times\) 88 & 83.5\% \\
2020 & [PERSON] et al. [31] & ResNet-18 & TCN & 88 \(\times\) 88 & 85.3\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Existing methods with different architectures trained on the LRW dataset. (LRW: Lip Reading in the Wild). Figure 10: (**a**) Feedforward without the residual connection. (**b**) Feedforward with the residual connection. Residual Connection is a kind of skip-connection that learns residual functions with respect to the layer inputs rather than learn unreferenced functions. However, the datasets above are all targeted at English lip-reading, and we still need Chinese datasets. Therefore, we decided to use a self-made dataset to perform our experiment. Pinyin, often shortened to just pinyin, is the official romanization system for Standard Mandarin Chinese, consisting of basic phonemes (56), consonants (23), and simple vowels (36). This results in 413 potential combinations plus special cases. Given the four tones of Mandarin, there are around 1600 unique syllables. Moreover, in Mandarin, if the phonemes such as 'b'/'p'/'m', 'd'/'t'/'n', 'g'/'k'/'h', 'zh'/'ch'/'sh' are followed by the same final, they show no obvious difference visually in lip movement. They are shown in Figure 11.  
To make the experiment less challenging, we avoid choosing Mandarin characters with the same Pinyin or similar Pinyin and collect more common and simple two-word characters in daily life for the time being. As soon as we make progress on the current dataset, further research for more complex Mandarin will be studied.