# Lip-Reading Classification of Turkish Digits Using Ensemble Learning Architecture Based on 3 DCNN
## 3 Material and Method
### Dataset Preprocessing  
Preprocessing involves preparing data for use in a machine learning method. This stage includes cleaning and editing the data and formatting the data for the model. Preprocessing is a part of machine learning that is usually performed before the data is analyzed, and includes the preparation of the data. In this study, two subsets was created, consisting of the front face and the lip region. The dataset was collected by recording 78 participants, aged between 20 and 55, speaking digits from 0 to 9 in Turkish. Each participant repeated each digit five times, resulting in a total of 3900 video clips. The recordings were made using an iPhone with a resolution of \(1080\times 1920\) and a frame rate of 30 fps. During the preprocessing phase of Dataset1 preparation, the videos were converted to frame format for convolution by making top and bottom cuts. Specifically, 420 pixels were cut from the top and bottom of the videos to create frames with a resolution of \(1080\times 1080\). The cutting area is shown in Figure 3.  
As shown in Figure 3, after cropping, the videos were resized to a frame format of \(1080\times 1080\). After this stage, the data was resized to \(224\times 224\) to reduce training time and computational costs associated with training of models. During this sampling process, the total number of frames of the video was sampled at equal intervals. By preserving the temporal context, multiple video data can be extracted by taking certain segments from long videos in a sequential order. However, this was not done for the digit dataset. This is due to the fact that the average length of each video for spoken digits is typically less than 1 s and the pronunciation of digits in Turkish is naturally short. In deep learning studies, data augmentation techniques are employed to increase the amount of data. One of the data augmentation methods for two-dimensional images is affine transformation methods, which include operations such as rotation, scaling, flipping, and transformation [31]. However, for video data, sampling methods from video frames are used instead. In this study, specific frames were sampled from each video file. The frames in the videos were rotated 10 degrees to the right and 10 degrees to the left as part of the data augmentation process, resulting in a threefold increase in the total amount of data. The number of data after data augmentation is shown in Table 3.  
Figure 3: Trimming during data preprocessing. Detection and Extraction of Lip Regions  
In the data set created for DataSet2, the lip region was extracted. The dataset was then extracted with RetinaFace [28] to focus only on the lip area. RetinaFace is a deep learning-based object detection algorithm for face detection and landmarks identification. Beyond verifying the existence of a face, the method is capable of detecting critical regions on the face (e.g., eyes, nose, mouth and lips) with high accuracy. RetinaFace provides a powerful solution for the precise detection of lip regions in particular. Using a ResNet-50 based architecture, RetinaFace is able to perform both face detection and detection of key points on the face at the same time. In this study, the center coordinates of the lips are determined using keypoints generated by RetinaFace to extract lip regions during the face detection process. During this process, the central coordinates of the lips of the face are taken as the basis and a square area is extracted around this central point, 112 pixels in both directions horizontally and 112 pixels in both directions vertically. Isolation of the lip region was performed by standardizing the frame sizes to \(224\times 224\) pixels, so that the original size of \(1080\times 1080\) pixels was reduced to \(224\times 224\) pixels. This dimensioning reduces the computational burden of the model while at the same time ensuring that the details of the lip movements are not lost. Figure 4 shows the extraction process from the lip region.