# Paraphrase Identification with Lexical, Syntactic and Sentential Encodings
## 3 LSSE Learning Model
### Syntactic Context Learning with R-GCNs  
We utilize R-GCNs to learn syntactic context. It can capture syntactic dependency structures naturally as well as word order because it allows the information to flow in the opposite direction of edges. For example, the sentence in the top of Figure 1, the word \"0.2-0.4\" modifies the word \"percent\". Let \(S\) be a sentence and \(w_{i}\) be the \(i\)-th absolute position word within the sentence. Let also \(G=(V,E)\) be a directed graph, where each node \(v_{i}\in V\) indicates the information of word \(w_{i}\), consisting of a word encoding \(\mathbf{x}_{i}\). BERT uses word pieces and not word embeddings. When \(w_{i}\) consists of several word pieces, we obtained the average value of all pieces corresponding to \(w_{i}\) and set it to the \(w_{i}\) embeddings. and its position encoding \(\mathbf{p}_{i}\) shown in Figure 2. We can define a matrix \(\mathbf{A}\in\mathbb{R}^{d\times n}\) where each column \(\mathbf{a}_{i}\in\mathbb{R}^{d}\) refers to the encoded node feature of \(v_{i}\), i.e., we sum up word and position encodings as lexical encoding, \(\mathbf{a}_{i}=\mathbf{x}_{i}+\mathbf{p}_{i}\). An edge from node \(v_{i}\) to \(v_{j}\) with a dependency relation type (label) \(l\in L\) is denoted by \(<\)\(v_{i}\), \(v_{j}\), \(l\)\(>\)\(\in E\), where \(L\) is a set of dependency relation types. Figure 3 illustrates dependency relations consisting of two information flows: from head to dependent and self-loop. Self-loop is to ensure that the representation of the encoded node feature at the \(k+1\)-th hidden layer can also be informed by its corresponding representation at the \(k\)-th hidden layer [28]. The propagation model for calculating the forward-pass update of a node \(v_{i}\) in a local graph can be given by:  
\[\mathbf{h}_{i}^{(k+1)} = f\left[\sum_{l\in L}\sum_{j\in N_{i}^{l}}\left(\frac{1}{c_{i}^{ \prime}}\mathbf{W}_{l}^{(k)}\mathbf{h}_{j}^{(k)}+\ \mathbf{b}_{l}^{(k)}\right)+\mathbf{W}_{0}^{(k)}\mathbf{h}_{i}^{(k)}\right], \tag{1}\]  
where \(\mathbf{h}_{i}^{(k)}\)\(\in\)\(\mathbb{R}^{d}\) is the hidden state of node \(v_{i}\) in the \(k\)-th layer of the neural network with \(d\) being the dimensionality of the hidden representations, especially the initial value of \(\mathbf{h}_{i}^{(0)}\) equals to \(\mathbf{a}_{i}\). \(N_{i}^{l}\) refers to the set of neighbor indices of node \(v_{i}\) under dependency label \(l\in L\). \(c_{i}^{l}\) shows a normalization constant [28]. It can either be learned or chosen in advance. We empirically set \(c_{i}^{l}\) to 2 in the experiments. \(\mathbf{W}_{l}^{(k)}\)\(\in\mathbb{R}^{d\times d}\) stands for the weight matrix and \(\mathbf{b}_{l}^{(k)}\)\(\in\)\(\mathbb{R}^{d}\) refers to the bias vector under label \(l\in L\) of the \(k\)-th hidden layer. We used 32 syntactic dependency relation types including _nsubj_ and _dobj_ provided by the Stanford parser for the first type of flows and their opposite direction types which would result in having 64 (32 \(\times\) 2) dependency labels. \(\mathbf{W}_{0}^{(k)}\)\(\in\)\(\mathbb{R}^{d\times d}\) indicates self-loop convolution weights and \(f\) refers to an activation function. We use the ReLU function. Equation (1) shows that it accumulates transformed feature vectors of neighboring nodes which depend on the relation type and the flow of an edge through a normalized sum. Motivated by the method of [PERSON] et al. [39], we also utilized a special gate mechanism. Our context learning model is given by:  
\[\mathbf{h}_{i}^{(k+1)} = f\left[\sum_{l\in L}\sum_{j\in N_{i}^{l}}s_{lj}^{(k)}\cdot \left(\frac{1}{c_{i}^{\prime}}\mathbf{W}_{l}^{(k)}\mathbf{h}_{j}^{(k)}+ \mathbf{b}_{l}^{(k)}\right)+\mathbf{W}_{0}^{(k)}\mathbf{h}_{i}^{(k)}\right], \tag{2}\]  
Figure 3: Dependency relations: “\(\mathbf{x}_{i}\)” and “\(\mathbf{p}_{i}\)” refer to the \(i\)-th word encoding and its position encoding, respectively.