# NAGNE: Node-to-Attribute Generation Network Embedding for Heterogeneous Network
## 3 Preliminary
### Node-to-Attribute Generation  
#### 4.3.1 Node Sequence Encoder  
Let \(\mathcal{L}=\{v_{1},v_{2},\ldots,v_{|\mathcal{L}|}\}\) be a sequence of nodes sampled from a network using attributed random walks; the corresponding nodes neighborhood sequence and node attribute type sequence are \(\mathcal{L}^{d}=\{\mathbf{D}_{1},\mathbf{D}_{2},\ldots,\mathbf{D}_{|\mathcal{L}|}\}\) and \(\mathcal{L}^{t}=\{t_{v_{1}},t_{v_{2}},\ldots,t_{v_{|\mathcal{L}|}}\}\), respectively. For each type of node, we apply a type-specific linear transformation to project the neighborhood of different types of nodes into the same latent factor space. For a node \(v_{m}\in A_{i}\) in node sequence \(\mathcal{L}\), we have  
\[h^{{}^{\prime}}_{v_{m}}=\mathbf{W}_{A_{i}}D_{m} \tag{8}\]  
where \(h^{{}^{\prime}}_{v_{i}}\in\mathbb{R}^{d^{{}^{\prime}}}\) is the projected latent vector of node \(v_{i}\), and \(\mathbf{W}_{A_{i}}\in\mathbb{R}^{d^{{}^{\prime}}\times n}\) is the parametric weight matrix for node type \(A_{i}\). Then, the sequence of node neighborhood \(\mathcal{L}^{d}=\{\mathbf{D}_{1},\mathbf{D}_{2},\ldots,\mathbf{D}_{|\mathcal{L}|}\}\) is projected as  
\[\mathcal{L}^{{}^{\prime}}=[h^{{}^{\prime}}_{v_{1}},h^{{}^{\prime}}_{v_{2}}, \ldots,h^{{}^{\prime}}_{v_{|\mathcal{L}|}}] \tag{9}\]  
where \(\mathcal{L}^{{}^{\prime}}\) is an \((|\mathcal{L}|,d^{{}^{\prime}})\) matrix. To capture the global topology information over sequences, NAGNE uses an encoder to encode the sequence of the node neighborhood vector \([h^{{}^{\prime}}_{v_{1}},h^{{}^{\prime}}_{v_{2}},\ldots,h^{{}^{\prime}}_{v_{| \mathcal{L}|}}]\), as mentioned in Transformer. The encoder is composed of a stack of \(L\) identical layers. Each layer has four sub-layers. The output of each layer is obtained from the output of the previous via four sub-layers as follows:  
\[\mathbf{Z}^{l}=RL\circ F\circ RL\circ MH(\mathbf{Z}^{l-1},\mathbf{Z}^{l-1}) \tag{10}\]  
where \(\mathbf{Z}^{l}\in\mathbb{R}^{|\mathcal{L}|\times d}\) denotes the hidden units of the \(l_{th}\) layer and, particularly, \(\mathbf{Z}^{0}\) is the node sequence neighborhood projecting \(\mathcal{L}^{{}^{\prime}}\); \(RL\), \(F\) and \(MH\), respectively, denote the sub-layers of residual connection plus layer normalization, feed-forward network and multi-head attention; and \(\circ\) denotes the composition operator between two functions. \(RL\) and \(F\) map a vector to another vector and we refer to reader to Transformer for detailed definitions. Multi-head attention allows the model to jointly attend to information from different representation sub-spaces, which is defined as follows:  
\[MH(\mathbf{Z},\mathbf{Z})=\varphi(head_{1},\ldots,head_{k}) \tag{11}\]where \(\varphi\) is a linear projection function, and  
\[head_{i}=Attention(\mathbf{ZW}_{i}^{1},\mathbf{ZW}_{i}^{2},\mathbf{ZW}_{i}^{3}) \tag{12}\]  
where \(\mathbf{ZW}_{i}^{1}\), \(\mathbf{ZW}_{i}^{2}\), \(\mathbf{ZW}_{i}^{3}\) are transformation matrices for head \(i\), and  
\[Attention(\mathbf{ZW}_{i}^{1},\mathbf{ZW}_{i}^{2},\mathbf{ZW}_{i}^{3})=softmax \left(\frac{\mathbf{ZW}_{i}^{1}(\mathbf{ZW}_{i}^{2})^{\intercal}}{\sqrt{d_{k}}} \right)\mathbf{ZW}_{i}^{3} \tag{13}\]  
The softmax function applied to the scaled dot-product attention scores normalizes them into a probability distribution over the input tokens. It is mathematically defined as  
\[softmax(\mathbf{x})_{j}=\frac{\exp(x_{j})}{\sum_{k=1}^{K}\exp(x_{k})} \tag{14}\]  
Here, \(\mathbf{x}\) is a vector containing attention scores before normalization, \(x_{j}\) is the \(j\)-th element of \(\mathbf{x}\), and \(K\) is the total number of elements in the vector. By exponentiating and normalizing each score, the softmax function ensures that the output is a valid probability distribution, with all elements between 0 and 1 and summing up to 1.