# Neuro-Symbolic Word Embedding Using Textual and Knowledge Graph Information
## 5 Conclusions and Future Work  
A method for constructing word embeddings using symbolic graphs with two properties was proposed. Our approach sequentially trains the syntactic and semantic relationships, as well as external knowledge using a GCN. The proposed method outperformed baseline models in word embedding benchmarks. Moreover, the performance of the model was improved when using the suggested word embeddings in each model for the extrinsic tasks. However, the proposed word embeddings still appeared to exhibit OOV problems. In future work, we will develop a graph representation containing nodes at the subword level and an efficient graph-based model.  
Conceptualization, software, investigation, methodology, writing--review and editing, writing--original draft: D.O., J.L.; investigation, validation, supervision, resources, project administration, and funding acquisition: H.L. All authors have read and agreed to the published version of the manuscript.  
This work was supported by an Institute of Information & communications Technology Planning & Evaluation (IITP) grant, funded by the Korean government Ministry of Science and ICT (MSIT) (No. 2020-0-00368, A Neural-Symbolic Model for Knowledge Acquisition and Inference  
\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Model** & **EM-Dev (\%)** & **F1-Dev (\%)** & **EM-Test (\%)** & **F1-Test (\%)** \\ \hline BiDAF with GloVe & 62.0 & 73.5 & 65.1 & 75.3 \\ \hline BiDAF with CN-GCN & **63.0** & **74.2** & **65.9** & **75.9** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance evaluation on SQuAD 1.1 with BiDAF. The performance is compared for initialization using GloVe or CN-GCN embedding. Bold indicates high performance.  
\begin{table}
\begin{tabular}{l c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{4}{c}{**Test Datasets**} & \multicolumn{2}{c}{**Concatenation of Test Datasets**} \\ \cline{2-5}  & **SE2** & **SE3** & **SE13** & **SE15** & **All** \\ \hline GAS (linear) with GloVe & 72.0 & 70.0 & 66.7 & 71.6 & 70.1 \\ GAS (concat) with GloVe & 72.1 & 70.2 & 67.0 & 71.8 & 70.3 \\ GAS\({}_{xtt}\) (linear) with GloVe & 72.4 & 70.1 & 67.1 & 72.1 & 70.4 \\ GAS\({}_{xtt}\) (concat) with GloVe & 72.2 & 70.5 & 67.2 & 72.6 & 70.6 \\ \hline GAS\({}_{xtt}\) (concat) with CN-GCN & **75.6** & **73.5** & **71.4** & **73.5** & **71.0** \\ \hline \hline \end{tabular}
\end{table}
Table 3: F1-score on fine-grained English all-word word sense disambiguation (WSD). Bold indicates high performance. The SE7 task was considered for all performances.  
\begin{table}
\begin{tabular}{l c c} \hline \hline
**Embedding** & **POS** & **NER** \\ \hline ELMo & 96.1 & 90.3 \\ \hline w/ Concat SemGCN embedding & 96.2 & 90.9 \\ \hline w/ Concat CN-GCN embedding & **97.0** & **91.2** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Performance when initialization was performed with each word embedding in parts-of-speech (POS) and named entity recognition (NER) tasks. Bold indicates high performance.  
Techniques) and by the MSIT, Korea, under the ICT Creative Consilience program (IITP-2022-2020-0-0-01819) supervised by the IITP. Furthermore, this research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF), funded by the Ministry of Education (NRF-2021R1A6A1A03045425).  
**Institutional Review Board Statement:** Not applicable.  
**Informed Consent Statement:** Not applicable.  
**Data Availability Statement:** ConceptNet: [[https://conceptne.io/](https://conceptne.io/)]([https://conceptne.io/](https://conceptne.io/)), accessed on 11 August 2019.  
**Conflicts of Interest:** The authors declare no conflict of interest.