# VR Training System to Help Improve Photography Skills
## 4 Automatic Evaluation Models for Composition and Color
### Building the Composition Evaluation Model  
The VEN model [30] was used as the model for evaluating composition. The VEN model is a model that utilizes the structure of VGG16 [32], one of the CNNs for image classification. The structure of the VEN model is shown in Figure 17.  
VGG16 is an architectural model that includes two blocks of \"convolutional layer, convolutional layer, pooling layer\" repeated twice; three blocks of \"convolutional layer, convolutional layer, pooling layer\" repeated three times; and culminates in three fully connected layers. Essentially, it is a 21-layer neural network comprising 13 convolutional layers, 5 pooling layers, and 3 fully connected layers. In addition, while VGG16 produces 1000 outputs, the VEN model only generates a single composition score. As such, in reference [30], VGG16's three fully connected layers were removed, retaining only the feature extraction portion of the model. Subsequently, two new fully connected layers and a single output layer were added to achieve the desired outputs. By adjusting the number of units in the fully connected layer to align with the fewer number of outputs, we succeeded in designing a model that is less complex than the original VGG16. Next, we explain the learning method of the VEN model. During the training, we used a framework called a Siamese network [33], as shown in Figure 18. Here, a Siamese network is a mechanism that compares the outputs of two input images through the same network with shared weights. More specifically, the input is a pair of images (image \(I_{i}\), image \(I_{j}\)) with composition ranking, and the output corresponds to the composition score of the input images \((f(I_{i}),\ f(I_{j}))\). The loss function is defined as follows based on [PERSON] et al. [34]. \[loss\big{(}I_{i},\ I_{j}\big{)}=max\big{\{}1+f\big{(}I_{j}\big{)}-f(I_{i}) \big{\}} \tag{3}\]  
Figure 17: Structure of the VEN model. Figure 18: Siamese network architecture. However, in Equation (3), image \(I_{i}\) is assumed to have a composition superior to that of image \(I_{j}\). Next, we describe the learning results of the VEN model. In this study, we conducted pre-training using the CPC dataset mentioned in the previous section. Then, we fine-tuned the model by dividing the created 13,000 pairs of images into a 9:1 ratio (training:validation). Figure 19 shows the transition of the VEN model's loss function and swap error [35]. Here, the swap error is the ratio of pairs that were mistakenly ranked as a result of inference for paired images with ranking. Continuing, we evaluated the created composition evaluation model using the FLMS dataset [36]. The FLMS dataset was created to evaluate the performance of image cropping tasks. Although this dataset consists of photos taken of the real world, we believe it is desirable to be able to appropriately evaluate composition even with real photos. The dataset includes 500 images, each with 10 correct labels (coordinates of rectangular areas). In previous studies [35; 37; 38], IOU and displacement error were used as evaluation metrics for cropping tasks. Therefore, we decided to use these evaluation metrics in this study as well. The two evaluation metrics are defined as follows. \[IOU=\frac{Area^{gt}\cap\ Area^{pred}}{Area^{gt}\cup\ Area^{pred}} \tag{4}\]  
\[Displacement\ Error=\sum_{k\in\{boundaries\}}\frac{\left|\left|B_{k}^{gt}-B_{k}^ {pred}\right|\right|}{4} \tag{5}\]  
where \(Area^{gt}\) represents the correct bounding box, \(Area^{pred}\) represents the predicted bounding box, and \(B_{k}\) represents the normalized boundary conditions. Finally, we show the results of the evaluation using the FLMS dataset [36] in Table 2. From the results, we confirmed that sufficient accuracy was achieved even in cropping tasks with real photos.