# Paraphrase Identification with Lexical, Syntactic and Sentential Encodings
## 5 Conclusions  
We focused on the problem that how to interpret a diversity of context information as relevant features and proposed an approach by leveraging a variety of features with a neural-based learning model. For syntactic encodings, our LSSE model incorporates word encoding, position encoding, and full dependency structures into a unified framework, R-GCNs. By utilizing the hidden states obtained by the R-GCNs as well as lexical and sentential encodings by BERT, our model learns contextual similarity between sentences. The experimental results by using two datasets showed that our model attained at 86.3% accuracy and 89.9% F1-score in MRPC, and 90.6% accuracy in QQP data which are comparable to the related work on paraphrase identification methods. Moreover, throughout the ablation test, we found that the combination of position encoding and syntactic features contributes to performance improvement.  
There are several interesting directions for future work. We should be able to obtain further advantages in efficacy in our syntactic embeddings obtained by the R-GCNs model. We empirically examined that the best performance is achieved with a two-layer model, while R-GCNs with more layers can be considered to capture richer neighborhood information of a graph. [PERSON] et al. focused on this problem and proposed a densely connected graph convolutional network that introduces residual connections, dense connectivity, and graph attention techniques [29]. They reported that the model attained at the current state-of-the-art neural models in the English-German and English-Czech translation tasks. This is definitely worth trying with our LSSE learning model.  
As we mentioned in Section 4.5, we found that more effective knowledge extraction improves the overall performance of paraphrase identification. Our model utilized BERT_base model for lexical and sentential encodings and applied it to two domain-specific data, MRPC and QQP. However, the BERT pre-training model is to learn general domain-independent knowledge. In the phase of fine-tuning, the model learns by using these domain-specific data which causes difficulty to estimate optimal parameters. Moreover, [PERSON] et al. reported that BERT is unstable when a training datasetwith fine-tuning is small [47]. One approach is to develop a knowledge transfer technique which is some empirical work along these lines in the deep learning field [64]. This is a rich space for further exploration.  
The following are available online at [[http://www.mdpi.com/2076-3417/10/12/4144/s1](http://www.mdpi.com/2076-3417/10/12/4144/s1)]([http://www.mdpi.com/2076-3417/10/12/4144/s1](http://www.mdpi.com/2076-3417/10/12/4144/s1)). Tables 2, 4, 5, 6.  
Conceptualization, F.F., and S.X.; methodology, S.X. and F.F.; software, S.X.; validation, S.X., F.F., X.S., and J.L.; investigation, S.X. and F.F.; writing--original draft preparation, S.X.; writing--review and editing, F.F., X.S., J.L., Y.S., and H.N.; supervision, F.F., and X.S.; funding acquisition, F.F. All authors have read and agreed to the published version of the manuscript.  
This work was supported by the Grant-in-aid for JSPS, Grant Number 17K00299, and Support Center for Advanced Telecommunications Technology Research, Foundation.  
The authors declare no conflict of interest.