# Semantic Representation Using Sub-Symbolic Knowledge in Commonsense Reasoning +
## 4 Experiments
### Baselines
#### 4.3.3 Graph Path Learning Module  
With the ACP-ARG graph from the graph integration and pruning module, the graph path learning module initializes the concept node vector as the sum of the concept embedding using GloVe [35] and absolute position embedding. A relation encoder is first used to encode the connection between the two concepts into a distributed representation for the model to recognize the explicit path of the ACP-ARG graph. The relation encoder recognizes the shortest path between two concepts and expresses the sequence as a relation vector by using a Gated Recurrent Unit (GRU) [36]. The equation representing the relation is depicted as follows:  
\[\overrightarrow{p_{t}}=\text{GRU}_{f}(\overrightarrow{p_{t-1}},sp_{t}), \overleftarrow{p_{t}}=\text{GRU}_{g}(\overleftarrow{p_{t+1}},sp_{t}) \tag{2}\]  
where \(sp_{t}\) is the shortest path of the relation between two nodes. The final relation encoding \(r_{ij}\) between concepts \(i\) and \(j\) is the concatenation of the final hidden states from the forward and backward GRU networks, which are represented in the Equation (3).  
\[r_{ij}=[\overrightarrow{p_{n}};\overleftarrow{p_{0}}] \tag{3}\]  
In order to inject this relation information into the conceptual representation, the AMR-CN reasoning model follow the idea of relative position, including [37; 38], which introduces an attention scoring method based on the conceptual representation and the relation representation.  
To calculate the attention score, the model is divided the relation vector \(r_{ij}\) passed from the linear layer into forward relation encoding \(r_{i\to j}\) and backward relation encoding \(r_{j\to i}\), as follows:  
\[[r_{i\to j};r_{j\to i}]=W_{r}r_{ij} \tag{4}\]  
where \(W_{r}\) is the parameter matrix. This split renders the model to consider the bi-directionality of the path. Thereafter, the model computes the attention score considering the concepts and their relations. Note that \(c_{i}\) and \(c_{j}\) are the concept embedding. The equation is presented below:  
\[\begin{split} s_{ij}&=f(c_{i},c_{j},r_{ij})\\ &=(c_{i}+r_{i\to j})W_{q}^{\top}W_{k}(c_{j}+r_{j\to i})\\ &=c_{i}W_{q}^{\top}W_{k}c_{j}+c_{i}W_{q}^{\top}W_{k}r_{j\to i}\\ &+r_{i\to j}W_{q}^{\top}W_{k}c_{j}+r_{i\to j}W_{q}^{\top}W_{k}r_{j \to i}\end{split} \tag{5}\]  
The first term in the last line of Equation (5) is the original term in the vanilla attention mechanism, which includes the pure content of the concept. The second and third terms capture the bias of the relation concerning the source and target, respectively. The last item represents universal relation bias. As a result, the computed attention score updates the concept embedding while maintaining a fully connected communication [15]. Therefore, the concept-relation interaction can be injected into the concept node vector. The resulting conceptual representations are aggregated into the entire graph vector and fed into the transformer layer to model the interaction between the AMR and ConceptNet conceptual representations.  
The major advantage of this relation-enhanced attention mechanism is that it provides a fully connected view of input graphs using the relation multi-head attention mechanisms. By integrating two different concept types from the AMR graph and ConceptNet into a single graph, the model globally recognizes which path has high relevance to the question during the interpretation.