# VR Training System to Help Improve Photography Skills
## 4 Automatic Evaluation Models for Composition and Color
### Building the Color Evaluation Model  
In this study, we used the Gated CNN model [39] for evaluating color. The structure of the Gated CNN model is shown in Figure 20. This model equally divides the input image (224 \(\times\) 224 \(\times\) 3) into patches (16 \(\times\) 16 \(\times\) 3), creating five patches for each set, which include the center patch (16 \(\times\) 16 \(\times\) 3) and its four neighboring patches (16 \(\times\) 16 \(\times\) 3). These are sequentially fed into the model. Including not only the center patch but also the neighboring patches allows for the learning of spatial positional information between patches. In [PERSON] et al. [39], training was performed with 10 sets extracted at random, but in this study, we used all possible patch sets (144 sets) for learning.  
In this context, the ResNet depicted in Figure 20 does not follow the standard 152-layer network structure. Instead, it adopts a structure akin to what is shown in Figure 21. Similar to the conventional ResNet, this model utilizes the shortcut connection mechanism, which allows for the direct addition of input from the preceding layers to the succeeding layers. This mechanism effectively mitigates the issue of gradient vanishing, a common problem in deep neural networks.  
Next, let us discuss the Gated CNN block in red in Figure 20. The structure of the Gated CNN block is illustrated in Figure 22.  
Figure 21: ResNet network architecture.  
Figure 20: Gated CNN model structure.  
In the Gated CNN model, the activation function for the convolutional layer diverges from the traditional ReLU function and instead employs the Gated Activation [40]. The definition of this activation function is as follows:  
\[f=tanh\Big{(}W_{f}\ast x_{f}\Big{)}\odot sigmoid\big{(}W_{g}\ast x_{g}\big{)} \tag{6}\]  
Here, \(\odot\) represents element-wise multiplication (Hadamard product), which calculates the product of matrix elements; \(x_{f}\) and \(x_{g}\) represent the feature maps divided into two from the output of the convolutional layer; and \(\ast\) represents the convolution operator. Equation (6) is also used in models such as WaveNet [41], which is a leading model in speech recognition and synthesis technology.  
In the above model, the output is composed of two harmony probabilities: the harmony probability \(p_{C}(x_{v})\) within the center patch, and the harmony probability \(p_{N}\Big{(}x_{v},x_{N(v)}\Big{)}\) between the center patch and the neighboring patches. Therefore, the overall harmony probability for the image was defined in [PERSON] et al. [39] as follows:  
\[P(X)=\frac{1}{K}\sum_{v\in K}\Big{(}\mu_{C}\cdot p_{C}(x_{v})+\mu_{N}\cdot p_{ N}\Big{(}x_{v},x_{N(v)}\Big{)}\Big{)} \tag{7}\]  
where \(K\) is the number of center patches, and \(\mu_{C}\) and \(\mu_{N}\) are hyperparameters. In this study, we followed [PERSON] et al. [39] and set \(\mu_{C}\) = 0.1 and \(\mu_{N}\) = 0.9. The loss function used was binary cross entropy.  
Next, we created a color dataset. In this study, we used annotated data that were rated on a five-point scale for color, and adopted the data from the two individuals who had the highest value of [PERSON]'s weighted Kappa coefficient [22] (Kappa coefficient: 0.545). For the five-point rating of annotated data, we created a histogram and converted it into a binary classification of 'Good' and 'Bad'.  
Finally, we present the learning results for the Gated CNN model. In this study, we divided 500 photo data into an 8:1:1 ratio, and for the training data, we performed data augmentation by rotating the images by 90 degrees, 180 degrees, and 270 degrees. We show the accuracy for the validation data and test data in Table 3. From the results, it seems that a relatively sufficient accuracy was obtained as an evaluation function of the training system, even in the color task.  
Figure 22: Structure of the Gated CNN block.