# Application of Convolutional Neural Networks in Visual Feedback of Movable Camera Mounting Control
## 2 Materials and Methods
### Dataset Preparation  
In this scenario, the operator had to always have eye contact with the screen for surveillance reasons. The data used for training should incorporate this information. This feature limits the usage of publicly available datasets. Usually, the head orientation is variable with an inline gaze, or the head orientation is constant, and the gaze is responsible for the desired action (e.g., mouse movement). In this project, none of the above applies. The gaze is pointed in the direction of a computer screen, whereas movements of the head should dictate the desired action. Due to the unique character of this problem, a new dataset was collected. First, a set of 17 classes directly connected with different head orientations was proposed. At first, the whole grid of 25 possible orientations was considered. However, the membership of certain images in a given class was ambiguous because of similarities between them. Therefore, a few classes that were irrelevant in terms of the control signal (e.g., HR, ST) were omitted. The set of examples of images representing all orientations is presented in Figure 2, whereas the corresponding output classes are gathered in Table 1.  
Head orientation is described as a pair of output numbers: (Horizontal orientation, Vertical orientation). For example, the head orientation depicted in the lower-left image in Figure 2 is denoted as (HR, HD), which represents the horizontal hard right orientation and the vertical hard left orientation. For each direction, five different orientations were distinguished. A total of 71 volunteers were asked to collect a set of 17 pictures, each similar to the example set. The volunteers had been instructed to keep the orientation of the head comfortable. They were asked to collect images at the center orientation, hard directional orientations, and slightly directional orientations in between the two previous ones. In each image, both eyes had to be visible, and the person should look in the direction of the screen. In particular, each image collection was acquired by a different non-professional camera with different final resolution and aspect ratio of the image. This was intentional, as the final application should estimate the head orientation on image frames from different amateur sources, e.g., web cameras. The resolution varied from 352 \(\times\) 480 px to 4032 \(\times\) 3024 px. Due to the wide variety of characteristics of the image data, a prior unification of the size was required. The coarse CNN (denoted in Figure 1) was used to this end. The main goal of coarse estimation was to detect the head bounding box. The coarse estimation \begin{table}
\begin{tabular}{c c c c} \hline \hline \multicolumn{1}{c}{**Horizontal**} & \multicolumn{2}{c}{**Vertical**} \\ \hline
**Head Orient.** & **Output Class** & **Head Orient.** & **Output Class** \\ \hline Hard Left (HL) & 0 & Hard Top (HT) & 0 \\ \hline Slightly Left (SL) & 1 & Slightly Top (ST) & 1 \\ \hline Center (C) & 2 & Center (C) & 2 \\ \hline Slightly Right (SR) & 3 & Slightly Down (SD) & 3 \\ \hline Hard Right (HR) & 4 & Hard Down (HD) & 4 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of the proposed output data classes. Figure 2: Example image set with the 17 assigned output classes. output data consisted of four corners of the bounding box. The corners of the bounding box were subsequently used to crop the images. The example result of this part of the algorithm is presented as the first step in Figure 3.  
The cropped images were preprocessed to constitute a valid training set of fine CNN estimation, as presented in Figure 3. First, the unified size was experimentally chosen to be \(128\times 128\) px, and the color information was discharged. Second, the image dataset was augmented by transforming the original image frames with typical operations (contrast change, noise addition, lightning change, and mirror reflection). Third, the features of the image were scaled to the range \((0,1)\), to ensure proper training.