# Towards Robust Representations of Spatial Networks Using Graph Neural Networks
## 4 Methods
### Learning on Heterogeneous Graph Representations of Spatial Networks
#### 4.2.3 Neural Architecture  
The neural architecture we use in this article is hierarchical, learning on the graph structure using the semantic paths [23; 28]. In order to differentiate between the node types during learning for a particular task, we define a transformation matrix \(\mathbf{Q}\) to enable a projection of nodes to a different feature space based on their type. We describe this in Equation (1). \[\mathbf{h}_{i}^{\prime}=\mathbf{Q}_{\mathcal{P}i}\cdot\mathbf{h}_{i} \tag{1}\]  
Here, \(\mathbf{h}_{i}\) refers to the original feature of node \(i\) and \(\mathbf{h}_{i}^{\prime}\) is the embedding derived for the same node. Next, we employ the concept of attention in [23] to assign importance to a node. In essence, given a node pair \((i,j)\) connected, we can learn how important either nodes will be to each other. Due to the heterogeneous nature of the graph, the importance of node \(i\) to node \(j\) may not be the same as the importance of node \(j\) to node \(i\). Hence, this is the non-symmetric nature of heterogeneous graphs. We define the attention \((a)\) between two nodes \((i,j)\) as  
\[e_{ij}^{\mathcal{P}}=a(\mathbf{h}_{i}^{\prime},\mathbf{h}_{j}^{\prime}). \tag{2}\]During training, we only calculate \(e_{ij}\) for nodes \(j\in\mathcal{N}_{i}\), where \(\mathcal{N}_{i}\) refers to the neighbourhood of node \(i\). We use the softmax activation function to normalize the attention values between node pairs to obtain the weight coefficient \(\alpha_{ij}^{\mathcal{P}}\)  
\[\alpha_{ij}^{\mathcal{P}}=softmax_{j}(e_{ij}^{\mathcal{P}})=\frac{\exp(\sigma (\mathbf{a}_{\mathcal{P}}^{\top}\cdot[\mathbf{h}_{i}^{\prime}||\mathbf{h}_{j}^ {\prime}]))}{\sum_{k\in\mathcal{N}_{i}^{\mathcal{P}}}\exp\left(\sigma(\mathbf{a }_{\mathcal{P}}^{\top}\cdot[\mathbf{h}_{i}^{\prime}||\mathbf{h}_{k}^{\prime}] )\right)} \tag{3}\]  
In Equation (3), \(\alpha\) refers to the softmax function, \(\alpha_{\mathcal{P}}\) is the node attention vector for a semantic path \(\mathcal{P}\), \({}^{\top}\) denotes transposition, \(\parallel\) is the concatenation operator. The embedding \(\mathbf{j}_{i}^{\mathcal{P}}\) for a node \(i\) from a specific semantic path \(\mathcal{P}\) can be derived from its neighbourhood, thus:  
\[\mathbf{j}_{i}^{\mathcal{P}}=\sigma\bigg{(}\sum_{j\in\mathcal{N}_{i}^{ \mathcal{P}}}\alpha_{ij}^{\mathcal{P}}\cdot\mathbf{h}_{j}^{\prime}\bigg{)} \tag{4}\]  
It has been demonstrated in [8; 23] that using more than one attention head will benefit model performance and stability. We employ \(K\) attention heads in this article, we compute the node attention \(K\) times and concatenate, extending Equation (4) as follows. \[\mathbf{j}_{i}^{\mathcal{P}}=\parallel_{k=1}^{K}\sigma\bigg{(}\sum_{j\in \mathcal{N}_{i}^{\mathcal{P}}}\alpha_{ij}^{\mathcal{P}}\cdot\mathbf{h}_{j}^{ \prime}\bigg{)} \tag{5}\]  
In addition to deriving the semantic-path based node embeddings, we seek to also determine embeddings for the semantic paths themselves using the attentional model. Following the proposal in [28], we define the weights for  
details about spatial objects, e.g., the class of streets such as residential streets and the use of buildings such as commercial buildings on OpenStreetMap. We construct the heterogeneous graph representations for these cities from OpenStreetMap using Algorithm 1. We consider two types of spatial objects: streets and buildings, and two types of relations: _intersects_ and _within_. The intersects relation denotes a spatial intersection between two objects and the within denotes objects that lie within \(x\) distance of another object. In this article, we set \(x\) to 500 m for all representations. The homogeneous graph representations are constructed for streets using only the _intersects_ relation between street segments. For buildings, we use only the _within_ relation between buildings, we set the distance to 500 m.  
All the graph representations used in this article are connected graphs, in cases where there are disconnected components, we select the largest graph component for our experiments.