## 3 Graph Neural Networks for Next-Generation Networks
### Backgrounds on Graph Neural Networks
#### 3.1.2 Message-Passing-Based Graph Neural Networks  
[PERSON] and [PERSON] [48] consider convolving the graph signal \(\mathbf{x}\) with a kernel of size \(K=1\). Additionally, they reduce the number of learned parameters to avoid overfitting. Accordingly, the convolutional operator is approximated as  
\[g_{\theta}(\mathbf{x})\approx\theta\Big{(}\mathbf{I}_{n}+\mathbf{D}^{-1/2}\mathbf{A}\mathbf{D}^{- 1/2}\Big{)}\mathbf{x}. \tag{2}\]  
By stacking \(k\) first-order convolutional layers together, we can approximate the \(k\)-order graph convolution. Although this is often referred to as the Graph convolutional network (GCN), many other GNNs are also designed with the same architectural pattern, which is to stack several sub-modules together. [PERSON] et al. [49] propose the generic framework Message-passing neural network (MPNN) to characterize this architectural pattern where these sub-modules could be described in one function:  
\[\mathbf{h}_{u}^{(t)} =\text{Update}^{(t)}\Big{(}\mathbf{h}_{u}^{(t-1)},\text{Agg}^{(t)} \Big{(}\Big{\{}\mathbf{h}_{v}^{(t-1)},\quad\forall v\in\mathcal{N}(u)\Big{\}} \Big{)}\Big{)} \tag{3}\] \[=\text{Update}^{(t)}\Big{(}\mathbf{h}_{u}^{(t-1)},\mathbf{m}_{u}^{(t-1)} \Big{)}, \tag{4}\]  
where \(\mathbf{h}_{u}^{(t)}\) is the _hidden_ embedding of node \(u\in V\) before being passed to layer \(t\), and \(\mathbf{h}_{u}^{(0)}=\mathbf{x}_{u}\) is the initial node features. In other words, an MPNN layer defines two important _differentiable_ functions (neural network as an example): _Update_ and _Agg_ (-regation). After the aggregation of the neighboring node embedding of node \(u\) into _message_\(\mathbf{m}_{u}^{(t-1)}\), the hidden embedding of node \(u\) is updated using its previous node embedding and the aggregated message. A simple example of an MPNN is illustrated in Figure 3. The GCN can also be regarded as an MPNN with the addition as _Agg_ and the learnable weighted sum as _Update_. In the same vein, we can consider the Graph attention network (GAT) [50] as an MPNN as well with the multi-head attention mechanism as _Agg_ and the concatenation as _Update_. We describe the updated message on node \(u\) after the \(t\)th layer of deep one-head attention GAT using equations below:  
\[\alpha_{u,v} =\frac{\exp\left(\text{LeakyReLU}(\mathbf{a}^{\top}[\mathbf{W}\mathbf{h}_{u} ||\mathbf{W}\mathbf{h}_{v}])\right)}{\sum_{v^{\prime}\in\mathcal{N}(u)}\exp\left( \text{LeakyReLU}(\mathbf{a}^{\top}[\mathbf{W}\mathbf{h}_{u}||\mathbf{W}\mathbf{h}_{v^{\prime}}]) \right)}, \tag{5}\] \[\mathbf{m}_{u} =\sum_{v\in\mathcal{N}(u)}\alpha_{u,v}\mathbf{h}_{v},\] (6) \[\mathbf{h}_{u}^{(t)} =[\mathbf{h}_{u}^{(t-1)}||\mathbf{m}_{u}^{(t-1)}], \tag{7}\]  
where \(||\) is the concatenation operator, LeakyReLU [51] is the non-linear activation function, \(\mathbf{a}\) and \(\mathbf{W}\) are learnable weights. For the sake of clarity, we omit superscript \((t-1)\) from \(\alpha_{u,v}\), \(\mathbf{a},\mathbf{W},\mathbf{h}\) and \(\mathbf{m}\) in Equations (5) and (6). The extension to multi-head attention [52] is straightforward.  
Overall, the main difference between spectral-based GCNs and MPNNs is their design patterns. While spectral-based GCNs provide a rigorous formulation for graph convolution, MPNNs simplify the computation, enable modularization, and facilitate solution design. In the following discussion, all GCNs are designed based on these two frameworks. Without further clarification, the term GCN refers to spatial-based GCN. Until now, GNNs continue to be a highly active research direction which still presents several challenges in both theoretical and technical aspects. However, in this section, and throughout this survey, our focus is exclusively on their applications in concrete networking problems and within the context of the Digital twin (DT). For a more in-depth exploration of challenges and current progress in this research field, we recommend readers to refer to [53; 54; 55; 56].