# Neuro-Symbolic Word Embedding Using Textual and Knowledge Graph Information
###### Abstract  
The construction of high-quality word embeddings is essential in natural language processing. In existing approaches using a large text corpus, the word embeddings learn only sequential patterns in the context; thus, accurate learning of the syntax and semantic relationships between words is limited. Several methods have been proposed for constructing word embeddings using syntactic information. However, these methods are not trained for the semantic relationships between words in sentences or external knowledge. In this paper, we present a method for improved word embeddings using symbolic graphs for external knowledge and the relationships of the syntax and semantic role between words in sentences. The proposed model sequentially learns two symbolic graphs with different properties through a graph convolutional network (GCN) model. A new symbolic graph representation is generated to understand sentences grammatically and semantically. This graph representation includes comprehensive information that combines dependency parsing and semantic role labeling. Subsequently, word embeddings are constructed through the GCN model. The same GCN model initializes the word representations that are created in the first step and trains the relationships of ConceptNet using the relationships between words. The proposed word embeddings outperform the baselines in benchmarks and extrinsic tasks.  
neuro-symbolic; graph convolutional network; word embedding; dependency parsing; semantic role labeling; ConceptNet; external knowledge +
Footnote †: journal:  
In this study, word embeddings are constructed sequentially through the neuro-symbolic learning of textual and knowledge graph information. A new graph representation is obtained to understand the syntactic and semantic relationships of the sentences. The example in Figure 1 depicts the graphs of the dependency parsing (DP) [14] and semantic role labeling (SRL) [9] of \"hot spot uses technology developed in the military for tank and jet fighter tracking.\" The DP represents all syntactic trees of a sentence for the syntax-aware model. The SRL enables the model to understand the predicate-argument relationships within sentences. In our proposed method, these two graphs improve the word embeddings through a graph convolutional network (GCN) [15; 16]. As word embeddings with symbolic information of the textual graph do not understand the relationships that are defined in the external knowledge graph, the model is trained using ConceptNet. The same architecture as that of the GCN is used in this process, and the word embeddings that are created by the first method are initialized.  
The contributions of this study are summarized as follows:  
* The proposed model sequentially trains on the symbolic information of the textual and external knowledge graphs to construct meaningful word embeddings.
* The proposed word embeddings ([[https://drive.google.com/file/d/1](https://drive.google.com/file/d/1) XXPYj47 onzAx--DVvslgib0 HvzGd7 wBR/view?usp=sharing]([https://drive.google.com/file/d/1](https://drive.google.com/file/d/1) XXPYj47 onzAx--DVvslgib0 HvzGd7 wBR/view?usp=sharing), accessed on 21 August 2022) outperform those of previous methods on word embedding benchmarks.
* Our word embeddings further improve the model performance in extrinsic tasks (parts-of-speech tagging (POS) [1], named entity recognition (NER) [17], word sense disambiguation (WSD) [18; 19; 20; 21; 22], and the Stanford Question Answering Dataset (SQuAD) 1.1) [23].