# Semantic Scene Graph Generation Using RDF Model and Deep Learning
## 4 System Implementation and Testing  
The test dataset used in this study for visual relation detection [6] was composed of 5000 images. The experimental dataset had a total of 37,993 relations, 100 objects and 70 predicates. Duplicate relations with the same subject-predicate-object were removed, regardless of the image. Therefore, 4000 images and 6672 relations were used; among which 1000 images and 2747 relations were used as test data. Eighty-six <Subject, Object> pairs that were not used in training were used to predict the relations. The test was implemented using Windows 10, with a Nvidia GTX 2060 6 GB GPU, running Python 3.7.4 and PyTorch 1.5.1 to conduct training. All the training processes took place in the same environment. We combined CNN and RNN models to implement the proposed method. The ResNet-101 architecture using ImageNet was used for the CNN model. The weights were initialized with values pre-trained on ImageNet. The following Table 3 shows the Top-K error index tested using ImageNet. We compared our method with [16] as the SOTA image recognition model [16] improved performance by using a skipped-connection technique. In our model, we replaced the last layer of ResNet-101, the FC layer, with the number of \(t_{i}\) and performed fine-tuning. In the case of the ResNet-101 model, the learning rate was 0.001, reduced by 0.1 every 20 epochs. Top-1 error and Top-5 error indexes were used to evaluate image classification performance. Top-1 error is the error rate in which the top selection predicate by the model is not the correct answer. Top-5 error is the error rate without a correct answer among the top five categories predicted by the model. Although our perfo \begin{table}
\begin{tabular}{c c} \hline \hline
**Rule Type** & **Example** \\ \hline Season Inference & \
\begin{tabular}{c} Between March and May it is spring, while from June to August it is summer, and from September to November it is autumn, and from December to February it is winter \\ \end{tabular} \\ Location Inference &
\begin{tabular}{c} If the location is part of Seoul, the location is also part of South Korea \\ \end{tabular} \\ Time Inference & If the time is eight to twelve o’clock, it is morning \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Example of User-Defined Rules. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Architecture** & **TOP-1 Error** & **TOP-5 Error** \\ \hline Ours & 22.73 & 6.44 \\ ResNet-101 [16] & 21.75 & 6.05 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: CNN backbone Top-K Error. that of [16] for image recognition, we focused on predicting the relationships between image tags. The result of predicting the predicate using the <Subject, Object> pair after extracting the tag using the CNN is shown in Figure 6. In the case of the LSTM model, the learning rate was 0.005, the embedding size was 300, the hidden size was 128 and the num layer was set to 1. The test was conducted by setting the prediction of each relation such that it exceeded.05 probability when passing through the Sigmoid activation function during relation prediction. Precision, recall and F1-Score were used as evaluation criteria. We found that the proposed LSTM method had higher precision than the GRU method, as shown in Figure 6. Although the recall of the proposed method was lower than that of the GRU, its F1-Score was higher. An example of the proposed method is shown in Figure 7. The first image shows the triple <Subject, Predicate, Object> obtained through the LSTM. However, some cases such as the triple <chair, touch, table> are not semantically correct because the \"touch\" predicate uses \"person\" as its domain type. The <person, wear, horse> triple in the third picture is also a semantically incorrect expression because the type range of \"wear\" is limited to \"cloth.\" However, a predicate that is not related to an actual image can be created. This is because all possible predicates were created based on <Subject, Object> in the dataset.