## 4 Methodology
### Heterogeneous Graph Neural Networks  
Heterogeneous graph neural networks (H-GNNs) are used for feature fusion of multimodal data, mapping data into heterogeneous graphs, combining artificial rules and automatic algorithms to obtain meta-paths, using graph convolutional networks to propagate and aggregate information; and increasing virtual nodes fuse information from different meta-paths, use an attention mechanism to calculate weights, use virtual nodes to represent the fusion of multimodal data, expand the number of meta-paths, and improve the fusion effect of long-distance related neighbors.  
**Graph embedding.** This module describes how we convert each Twitter data into a heterogeneous graph node. Since different tweets contain different data types, each tweet contains a different number of nodes. A heterogeneous graph is constructed for all data \(G=(V,E)\), where \(V\) is the set of nodes and \(E\) is the set of edges. Each node \(v\in V\) has a type \(\phi_{v}\in\Gamma_{v}\), and each edge \(e\in E\) has a type \(\phi_{e}\in\Gamma_{e}\). \(\Gamma_{v}\) and \(\Gamma_{e}\) are a node type set and an edge type set, respectively. Each node \(v\) also has a feature vector \(x_{v}\in R^{d}\), where \(d\) is the feature dimension.  
**Generate reformed graph.** Our dataset implements fine-grained labeling, that is, labeling all modalities of each piece of data, and there is also a labeling for the entire piece of data. The virtual node is defined as \(v^{*}\), and there is an edge connected to all modal nodes, and the type of this edge is \(\phi_{e^{*}}\). The state information is fused and passed to other nodes, and its eigenvector \(x_{v}\), can be initialized to the average value of all modal node eigenvectors or other methods.  
We define a meta-path \(P\) as a sequence of adjacent edge types, such as \(P=\phi_{e1}\rightarrow\phi_{e2}\rightarrow\cdots\rightarrow\phi_{ek}\), which can be used to represent a semantic relationship between two nodes.  
We define a heterogeneous graph neural network \(H\) as a function, which can map the feature vector \(x_{v}\) of each node \(v\) to a low-dimensional vector \(h_{v}\in R^{k}\), where \(k\) is the embedding dimension, and its goal is to enable \(h_{v}\) to capture the structural and non-structural information of \(v\).  
**Aggregation.** The graph neural network has a unique aggregation idea; that is, nodes are aggregated according to the meta-path [53]. The fusion of different types of data is achieved by sequentially aggregating adjacent nodes to itself. We assume that the heterogeneous graph neural network \(H\) consists of \(L\) layers, and each layer is a process of information aggregation and transformation. Level \(l\) can be expressed as  
\[h_{v}^{(l)}=f^{(l)}\Big{(}x_{v},\Big{\{}h_{u}^{(l-1)}:u\in N_{v}\Big{\}}\Big{)} \tag{4}\]  
where \(h_{v}^{(l)}\) is the embedding vector of node v in layer \(l\), \(f^{(l)}\) is the information aggregation and transformation function of layer \(l\), and \(N_{v}\) is the set of neighbor nodes of node v, including the virtual node \(v^{*}\). We can define different neighbor node sets \(N_{v}^{P}\) according to different meta-paths \(P\); for example, \(N_{v}^{P}=\Big{\{}u:u\in N_{v},\phi_{(u,v)}=P\Big{\}}\) represents the set of neighbor nodes connected with node \(v\) by meta-path \(P\).  
In this way, we can obtain the embedding vector \(h_{v}^{(l)}\) of each node in each layer, and the output \(h_{v}^{(L)}\) of the last layer is the final embedding vector for \(v\). We can use this embedding vector for tasks like multimodal sentiment analysis [52; 54].