# VR Training System to Help Improve Photography Skills
## 3 VR Photo Training System
### Aesthetic Photo Evaluation Model  
In this research, we used an aesthetic evaluation model based on a convolutional neural network (CNN) to assess photographs. An aesthetic evaluation model is a model designed to extract features, such as the composition, color scheme, and blur of an image, and to evaluate the quality of the photograph. From the perspective of accuracy and computational cost, we utilized the NIMA (neural image assessment) model [1]. The architecture of the NIMA model is shown in Figure 5. The MobileNet [19] shown in Figure 5 is one of the architectures used in image classification tasks, and it is known as a model that emphasizes maximizing accuracy while being lightweight.  
MobileNet employs a unique computational technique known as \"depthwise separable convolutions\", which provides a more streamlined alternative to traditional convolution operations. Regular convolutional operations apply convolutional filters to all input image channels (such as red, green, and blue in an RGB image), affecting spatial and channel convolutions simultaneously. However, depthwise separable convolutions partition this operation into two sequential steps, namely \"Depthwise Convolution\" and \"Pointwise Convolution\", thereby diminishing computational demands. To elucidate, Depthwise Convolution involves performing convolutions individually for each channel, meaning each filter interacts with just one channel. Following this, in Pointwise Convolution, the output from each channel via Depthwise Convolution is merged through a \(1\times 1\) convolution operation. Hence, after the spatial convolution in Depthwise Convolution, channel convolution is carried out in Pointwise Convolution. This sequential, rather than simultaneous, operation of spatial and channel convolutions curbs computational expenses. For instance, with a \(3\times 3\) filter size, 64 input channels, and 128 output channels, standard convolution operations would necessitate a total of 73,728 multiplications (\(3\times 3\times 64\times 128\)). In contrast, depthwise separable convolutions only require 576 multiplications (\(3\times 3\times 64\)) in the depthwise phase and 8192 (\(64\times 128\)) in the pointwise phase, culminating in 8768 operations in total. This stark contrast showcases the significant reduction in computational complexity. Considering the need for real-time feedback in this study, the NIMA model employing MobileNet, with its low computational burden, was deemed an appropriate choice.  
The NIMA model takes a single image as input and outputs a probability distribution of aesthetic scores. By calculating the weighted average of the outputted probability distribution, a binary classification is performed to determine if the image is aesthetically pleasing. Therefore, the loss function uses the EMD (Earth Mover's Distance) [20], which  
Figure 4: Eight different facial expressions.  
Figure 5: NIMA model.  
measures the distance between two probability distributions, rather than the typical Binary Cross Entropy. The EMD can be expressed with the formula:  
\[EMD(p,\ \hat{p})=\left(\frac{1}{N}\ \sum_{k=1}^{N}\left|CDF_{p}(k)-CDF_{\hat{p}}(k) \right|^{r}\right)^{\frac{1}{r}} \tag{1}\]  
On the left side of the equation, \(p\) and \(\hat{p}\) represent the probability distribution of the correct score and the predicted score, respectively. On the right side, \(N\) denotes the number of ordered classes, \(CDF\) represents the cumulative distribution function, and \(r\) represents the norm. In this research, based on [PERSON] and [PERSON] [1], we set \(r=2\).  
In this study, the evaluation was conducted as a binary classification problem based on previous research [1], so we followed that and used binary classification as the output of the photo aesthetic evaluation model. However, since the NIMA model uses the EMD loss to predict the histogram of scores, it would not be a problem to calculate the weighted mean and output it as it is with continuous values.