# MGKGR: Multimodal Semantic Fusion for Geographic Knowledge Graph Representation
## 3 Methods
### Two-Stage Multimodal Feature Fusion  
This paper combines the features from text, image, and structure by a two-stage multimodal feature fusion that incorporates Spatial Feature Fusion for Modality Enhancement and Multi-level Modality Feature Fusion Encoder. **Spatial Feature Fusion for Modality Enhancement** (SFF). The SSF module enhances text embedding \(\mathbf{H^{t}}\) and visual embedding \(\mathbf{H^{v}}\) by structural embedding \(\mathbf{H^{s}}\). Although the text and visual embedding exhibit different distributions with the structural embedding in the knowledge graph, their underlying correlations enable complementary feature integration. To leverage this underlying correlation, an adaptive feature fusion method is utilized. For the text features, a learnable parameter matrix \(\mathbf{W^{g}_{1}}\) is used to adaptively obtain relevant information from the structural features \(\mathbf{H^{s}}\), which is then fused with the text features \(\mathbf{H^{t}}\). For the visual features, another matrix \(\mathbf{W^{g}_{2}}\) is used to capture relevant information from the structural features and fuse it with the visual features \(\mathbf{H^{v}}\). As demonstrated in Equations (6) and (7). This method takes into account the underlying correlations between modalities, adaptively extracting structural features and fusing them with text and visual features. \[\mathbf{H^{gt}}=\mathbf{H^{t}}+\mathbf{W^{g}_{1}}\times\mathbf{H^{g}}, \tag{6}\]  
\[\mathbf{H^{sv}}=\mathbf{H^{v}}+\mathbf{W^{g}_{2}}\times\mathbf{H^{g}}, \tag{7}\]  
where \(\mathbf{H^{gt}}\) and \(\mathbf{H^{sv}}\) represent the text feature and image feature after fusing structural feature, respectively. To further mitigate the impact of distributional heterogeneity among different modal features during feature fusion, while also capturing the underlying common features across different modalities this paper utilizes contrastive learning to align the distribution of multimodal semantic embedding with the structural embedding of geographic entities. The contrastive learning adopts the InfoNCE loss function [41], as outlined in Equations (8) and (9). Specifically, by constructing positive and negative sample pairs, the method brings the structural features \(\mathbf{h^{g}_{i}}\) of entity \(i\) closer to its text features \(\mathbf{h^{t}_{i}}\) and visual features \(\mathbf{h^{v}_{i}}\) as positive pairs while pushing the text and visual features of other entities away as negative pairs. This contributes to a more uniform distribution of the samples, mitigates the impact of heterogeneity, and further enhances the model's ability to extract underlying common features. \[L_{ct}=\sum_{i}^{N}-\log\left(\frac{\exp\bigl{(}\cos\bigl{(}\mathbf{h^{g}_{i}},\bm {h^{t}_{i}}\bigr{)}/t\bigr{)}}{\exp\bigl{(}\cos\bigl{(}\mathbf{h^{g}_{i}},\mathbf{h^{t }_{i}}\bigr{)}/t\bigr{)}+\sum_{j}^{\left|N^{\prime}\right|}\exp\bigl{(}\cos \bigl{(}\mathbf{h^{g}_{i}},\mathbf{h^{t}_{j}}\bigr{)}/t\bigr{)}}\right) \tag{8}\]  
\[L_{cv}=\sum_{i}^{N}-\log\left(\frac{\exp\bigl{(}\cos\bigl{(}\mathbf{h^{g}_{i}},\bm {h^{v}_{i}}\bigr{)}/t\bigr{)}}{\exp\bigl{(}\cos\bigl{(}\mathbf{h^{g}_{i}},\mathbf{h^{v }_{i}}\bigr{)}/t\bigr{)}+\sum_{j}^{\left|N^{\prime}\right|}\exp\bigl{(}\cos \bigl{(}\mathbf{h^{g}_{i}},\mathbf{h^{v}_{j}}\bigr{)}/t\bigr{)}}\right) \tag{9}\]  
In the equations \(\mathbf{h^{t}_{i}}\in\mathbf{H^{t}},\mathbf{h^{v}_{i}}\in\mathbf{H^{v}}\), \(N\) represents the total number of samples, while \(N^{\prime}\) denotes the number of negative samples, and \(t\) is a hyperparameter. The function \(\cos(\cdot,\cdot)\) computes the cosine similarity between two embeddings. **Multi-level Modality Feature Fusion Encoder** (M-Encoder) integrates the enhanced text embedding \(\mathbf{H^{gt}}\) and visual embedding \(\mathbf{H^{sv}}\). The weak correlation between text and visual data in multimodal geographic knowledge graphs introduces modality heterogeneity, which could undermine the effective extraction of meaningful information during the feature fusion.