# Semantic Scene Graph Generation Using RDF Model and Deep Learning
## 3 Proposed System
### Creating a Deep Learning-Based Scene Graph  
Image tagging was performed using a CNN for image annotation in a previous study [10]. However, training images using a CNN makes it impossible to understand the semantics of objects. Furthermore, since the previous method made a direct SPARQL query on DBPedia ([[https://wiki.dbpedia.org/](https://wiki.dbpedia.org/)]([https://wiki.dbpedia.org/](https://wiki.dbpedia.org/))) for detecting relations between images tags, it was impossible to find a relation that was not already stored in DBPedia. Therefore, although we performed image tagging using a CNN, as in the previous study, our approach predicted relations between tags after vectorization and image tagging using a long short-term memory (LSTM) network, as well as gated recurrent units (GRU), an RNN model. The overall process of the proposed method is as shown in Figure 3.  
First, in our proposed method, tags are created for image objects using a CNN model. In particular, we used the ResNet-101 model [16] as the state-of-the-art (SOTA) CNN model for image tagging because it uses skip-connection techniques and has effective performance on ImageNet. The process for image tagging using ResNet-101 is as follows. First, the CNN model extracts the features of images using a filter and then stores them in various  
Figure 3: Proposed system. Figure 2: Resource description framework (RDF) and RDF schema for semantics. feature maps. The fully connected (FC) layer uses a value projected by the number of each class (tags) from each layer to classify the final image of the extracted feature map. In the output layer, the predicted value of the image tag t\({}_{i}\) is output using the Sigmoid activation function, shown in Equation (1) below:  
\[\sigma(x)=\frac{1}{1+e^{-x}} \tag{1}\]  
The Sigmoid function \(\sigma\) is used for outputting values for each class and is defined as follows in Equation (2):  
\[t_{i}=\sigma(Wx_{i}+b) \tag{2}\]  
where \(W\) is the FC layer value, and \(b\) is the bias term. Then, we adopt a binary cross-entropy loss function given as follows:  
\[\mathcal{L}_{1}=\sum_{i=1}^{N}\left(\sum_{j=1}^{M}y_{ij}log\left(t_{ij}\right) +\left(1-y_{ij}\right)log\left(1-t_{ij}\right)\right) \tag{3}\]  
where \(x_{i}\) is the input image, N is the number of image datasets, M is the number of tags, \(t_{ij}\) is a predicted value of a tag \(j\) for an image \(x_{i}\), and \(y_{ij}\) is the correct answer for the image \(x_{i}\). For example, \(y_{ij}=1\) indicates that the image \(x_{i}\) contains tag j. The process of detecting an object in an image using the CNN model is shown in Figure 4.  
The disadvantage of processing images using a CNN is that it becomes impossible to understand the semantics of the objects (tag words). To understand the meaning of words, techniques such as Word2 Vec [17], which expresses words as vectors, have been created. Furthermore, systems such as ConceptNet [18] have become available to help computers understand the relationships between words. ConceptNet is a semantic network designed to enable computers to understand the semantics of words used by people. It provides a 300-dimensional tag representation vector \(w_{i}\) per word embedded. \(w_{i}\) is a 300-dimensional vector that passes through the embedding layer in the image tag \(t_{i}\), represented as follows:  
\[w_{i}=word2 vec(t_{i})\left(w_{i}\in\mathcal{R}^{300}\right) \tag{4}\]  
In this paper, we propose an RDF model-based scene graph describing an image using triple sets of the form <Subject, Predicate, Object> (For convenience, the <Subject-Predicate-Object> triple is denoted as <Subj, Pred, Obj>. The <Subject, Object> pair is expressed  
Figure 4: Convolutional neural network (CNN) model for image tagging. as <Subj, Obj>.). Thus, we predict the predicate \(p_{j}\) using \(t_{subj}\) and, \(t_{obj}\), selected from the image tag set \(\{t_{1},t_{2},\cdots,t_{i}\}\). We use an RNN model, able to store previous data in a hidden state, to predict the sequence relationship of <Subj, Pred, Obj>.