# MGKGR: Multimodal Semantic Fusion for Geographic Knowledge Graph Representation
## 3 Methods
### Multimodal GeoKG Encoding  
The multimodal geographic knowledge graph incorporates diverse geographic data, including entity IDs, coordinates, attributes, and multimodal data such as text and images. The multimodal geographic knowledge graph is represented as \(G_{\text{geo}}=\{V,R,E,M\}\), where \(V=\{V_{\text{geo}},V_{\text{att}}\}\) denotes the set of geographic and attribute entities, \(R=\{R_{\text{adj}},R_{\text{att}}\}\) represents spatial and attribute relations, \(E=\{(s,r,t)\mid s,t\in V,r\in R\}\) is the set of triples, and \(M=\{M_{\text{text}},M_{\text{image}}\}\) includes the multimodal data. 1. Structural Feature Encoding  
The structural features in GeoKG can provide spatial topological information between geographic entities, as well as the relation feature between geographic entities and attribute entities. For the geographic knowledge graph \(G_{\text{geo}}\), MGKGR utilizes a KRL model to encode GeoKG and capture its structural features. Specifically, we obtain the initial embedding for each entity \(\mathbf{H^{0}}=\{\mathbf{{{}^{0}_{4}}},\mathbf{{{}^{0}_{2}}},\mathbf{{{}^{0}_ {3}}},\ldots,\mathbf{{{}^{0}_{n}}}\}\). These entity embeddings are then encoded through the KRL model. To effectively capture the spatial features in KG with multiple relations, we adopt RGCN [14] as the KRL model in MGKGR, which captures the structural features  
Figure 2: The framework of the proposed MGKGR. (**A**) Multimodal GeoKG Encoding module processes the multimodal data of multimodal GeoKG for effective encoding. (**B**) Two-Stage Multimodal Feature Fusion module integrates features from multiple modalities to generate the multimodal features of multimodal GeoKG. \(\mathbf{H^{g}}\) of the geographic knowledge graph by aggregating the neighboring node features \(\mathbf{h_{i}}\) of each entity. The training formula is presented in Equation (1). \[\mathbf{h_{i}^{l+1}}=\sigma\left(\sum_{r\in R}\sum_{j\in N_{i}^{r}}\frac{1}{|N_{i}^{ r}|}\mathbf{W_{i}^{l}}\mathbf{h_{j}^{l}}+\mathbf{W_{g}^{l}}\mathbf{h_{i}^{l}}\right), \tag{1}\]  
where \(N_{i}^{r}\) is the set of first-order neighbors of entity \(i\) under relation \(r\), and \(|N_{i}^{r}|\) is used to balance the influence of varying numbers of neighbors. \(\mathbf{h_{i}^{l}}\) is the \(l\)-th layer representation of node \(i\), with the final layer output representing the structural feature of the geographic knowledge graph, denoted as \(\mathbf{H^{g}}\). 2. Text Feature Encoding  
To capture the rich semantic information contained in the text, MGKGR employs a text encoder to encode the textual data. The text encoder is composed of the first \(L_{t}\) layers of BERT [36]. To more effectively leverage the capabilities of the Transformer encoder for better text feature representation, we adopted the concept from MKGFormer [35] by converting the link prediction task into a Masked Language Modeling (MLM) task. The predicted tail entity in the link prediction is treated as the token masked with [MASK] in the MLM task. Specifically, we organized the triples in the form presented in Equation (2). \[\text{Text}_{(e_{h},r,7)}=\left[\text{CLS}\right]e_{h}\,d_{e_{h}}\left[\text{ SEP}\right]r\,[\text{SEP}]\,[\text{MASK}]\,[\text{SEP}], \tag{2}\]  
where \(e_{h}\in V\), \(r\in R\), and \(d_{e_{h}}\in M_{\text{text}}\) represent the textual modality data corresponding to the head entity \(e_{h}\). [CLS] is a special token used to aggregate features from the entire input, while [SEP] is used to separate different parts of the input. The tail entity to be predicted is replaced by \([MASK]\), and the \"?\" in \(\text{Text}_{(e_{h},r,7)}\) represents the tail entity that needs to be predicted. This manner not only makes more effective use of the textual data of entities but also incorporates relations in a textual format, allowing for rich semantic relation expressions that enhance the ability of the model to capture and represent relationships.