# Neuro-Symbolic Word Embedding Using Textual and Knowledge Graph Information
## 3 Proposed Method  
In this study, the GCN [15; 16] is exploited to learn the graph representation for word embedding. The proposed model consists of two stages. First, the model trains a new graph representation that combines the DP and SRL graphs. Second, the model initializes the word representations that are constructed in the first process and learns the inner correlations within ConceptNet on the same structure. The graph is defined as \(G=(V,E,X)\), where the node set is \(|V|=n\), and \(E\) represents the edge set. \(X\in\mathbb{R}^{n\times d}\) means an input node with \(d\)-dimensionality. When nodes \(u\) and \(v\) exist, \(E(Edge)\) from \(u\) to \(v\) is labeled as \((u,v,l_{uv})\). Moreover, as information does not always need to propagate in one direction, the reverse direction \((u,v,l_{uv}^{-1})\) is included in \(E(Edge)\). The proposed model is depicted in Figure 2. The sentence \(s\) is composed of a set of words such as \(s=w_{0},w_{1},\ldots,w_{n}\). TG-GCN  
The text graph GCN (TG-GCN) performs training by combining the two graphs. Unlike in DP, the SRL information is predicted with a BIO tag on the sentence. Therefore, the defined tags are replaced with a new graph representation. For example, in Figure 1, the \"hot spot\" that is connected to \"uses\" is defined as hot (B-ARG0) and spot (I-ARG0). When this is represented as an edge, (\"uses\", \"hot\", B-ARG0), (\"uses\", \"spot\", I-ARG0) is expressed. Following the combination of the two graphs, the graph information is defined as \(TG=(V_{TG},E_{TG},X_{TG})\). In Equation (1), \(h_{v}^{k+1}\) is represented through the \(k\)-GCN layer with the newly created graph representation as the input, where \(h_{v}^{k+1}\) is \(h_{v}^{k+1}\in\mathbb{R}^{d}\). \[h_{v}^{k+1}=f(\sum_{u\in N_{+}(v)}(W_{l_{uv}}^{k}h_{u}^{k}+b_{l_{uv}}^{k})), \tag{1}\]  
where \(W_{l_{uv}}^{k}\in\mathbb{R}^{d\times d}\) and \(b_{l_{uv}}\in\mathbb{R}^{d}\) denotes the model parameters for the relationship between nodes. Furthermore, the adjacency set of \(N_{+}(v)\) (including \(v\) itself) and \(h_{v}^{k}\in\mathbb{R}^{d}\) is the representation of node \(v\) through layer \(k-1\). When using an external model to generate graph structures from the corpus automatically, it is possible to include incorrect edges and labels within the graph structure. Error propagation occurs during the training process of the models because manual tagging is not performed. To address this limitation, our method calculates the edgewise gating score, as proposed by [PERSON] and [PERSON] [31]. The edge-label gating mechanism score is applied to the node representation in Equation (1) of the GCN. The score is calculated  
Figure 2: Overview of the proposed method. The model sequentially trains symbolic graphs of different properties in two steps. independently for all connected edges of each node \(v\), and the difference between the edges is distinguished. The score \(g^{k}_{l_{uv}}\in\mathbb{R}\) is determined as indicated in Equation (2). \[g^{k}_{l_{uv}}=\sigma(\hat{W}^{k}_{l_{uv}}h^{k}_{u}+\hat{b}^{k}_{l_{uv}}), \tag{2}\]  
where \(\hat{W}^{k}_{l_{uv}}\in\mathbb{R}^{1\times d}\) and \(\hat{b}^{k}_{l_{uv}}\in\mathbb{R}\) is a learnable parameter. Furthermore, \(\sigma(\cdot)\) is a sigmoid function. Finally, \(h^{k+1}_{v}\) with the edgewise gating mechanism is expressed by Equation (3). \[h^{k+1}_{v}=f(\sum_{u\in N+(v)}g^{k}_{l_{uv}}\times(W^{k}_{l_{uv}}h^{k}_{u}+b^{ k}_{l_{uv}})) \tag{3}\]  
CN-GCN  
The word embeddings that are constructed by the TG-GCN only understand limited semantic relationships within the context. For example, the model merely understands whether the word \"apple\" is a company or food through the surrounding contexts \"eat\" or \"computer\", respectively. However, the model does not know whether \"apple\" is a hyponym for \"fruit\" The model can only understand such factual information using structured external knowledge. Therefore, our method creates meaningful word embeddings by learning the semantic relationships of ConceptNet.