# Neuro-Symbolic Word Embedding Using Textual and Knowledge Graph Information
## 2 Related Works  
Word Embedding  
Thus far, research on word embeddings has received substantial attention in NLP. The continuous bag-of-words (CBOW) and skip-gram (SG) [24] methods are based on language modeling using neural networks to represent words as real-valued vectors. However, these methods do not consider word co-occurrences in sentences. The global vectors for word representation (GloVe) [25] approach extends the use of global statistical information. FastText [26] uses subwords to solve problems such as out-of-vocabulary (OOV) words. Figure 1: Examples of dependency parsing (DP) and semantic role labeling (SRL). In DP, any dependencies between words in a sentence are represented in a graph structure. In SRL, arguments are labeled with BIO tags according to predicates. For example, for the predicate “use” and the linked argument “hot spot”, “hot” is B-ARG0 and “spot” is I-ARG0. (**a**) Example of dependency parsing. (**b**) Example of semantic role labeling. In recent years, language models have been proposed that represent words as different real-valued vectors according to the context. The Embeddings from Language Model (ELMo) [27] is an LSTM-based bidirectional language model, whereas bidirectional encoder representations from Transformers (BERT) [28] is a Transformer-based bidirectional language model. BERT is pretrained with a large corpus through masked-language modeling (MLM) and next sentence prediction (NSP) objectives. These contextual representation models have achieved high performance on various NLP tasks. However, as these methods do not learn syntactic information, the understanding of the relationships between words in a sentence is limited. In a study by [PERSON] and [PERSON] [7], the encoding of syntactic relationships between words demonstrated that functional similarity exhibited higher performance than topical similarity. Another study revealed improved performance when words were represented with syntactic information using second-order [29] and multi-order [30] dependencies. It is necessary to label a large corpus automatically when employing modes that use syntactic information. Therefore, these models exhibit errors that are propagated as a result of data that are constructed in this process. To address this issue, [PERSON] et al. [8] used an edge label gating mechanism to enhance the performance compared to previous models. However, models that learn only syntactic information experience difficulty in understanding semantic correlations such as external knowledge or the relations of predicate arguments among words. Semantic Role Labeling (SRL)  
SRL [9] uses the concept of the predicate-argument relation. Each predicate is connected to various arguments, and the linked arguments play a corresponding semantic role. For example, in the sentence \"[PERSON] gives [PERSON] a book\", \"gives\" is a predicate, whereas \"[PERSON]\", \"[PERSON]\", and \"a book\" are arguments. \"[PERSON]\" is classified as an agent from the arguments, and \"a book\" is an argument with the theme of \"give\" That is, when the user asks \"What is [PERSON] giving to whom?\" the answer of the model is \"a book\" Furthermore, the answer to the question \"Where is a book?\" is \"Susie\", so \"Susie\" corresponds to a goal among the types of arguments. We use the model provided by AllenNLP ([[https://demo.allennlp.org/semantic-role-labeling](https://demo.allennlp.org/semantic-role-labeling)]([https://demo.allennlp.org/semantic-role-labeling](https://demo.allennlp.org/semantic-role-labeling)), accessed on 11 August 2019) to construct the SRL graph. Dependence Parsing (DP)  
DP [14] is a method of representation in the syntax tree structure that defines the relations between words in a sentence. This method presupposes that a correlation exists between the words of a sentence that influence one another.