## 2 Method
### Multimodal Supervised Learning Network  
After obtaining entity encodings using an entity encoder, the entities are fine-tuned to incorporate multimodal information representations through multimodal supervised learning. Specifically, for the relationship and attribute information of entities, features are extracted using a feed-forward network, while graph structure information is acquired using GCN, and image information is extracted using ResNet. Relationship and Attribute Embedding: Since modeling relationships and attributes using GCNs might result in contaminated entity representations due to noise interference from neighbors [25], a simple feed-forward network is employed to map relationship and attribute features in a low-dimensional space:  
\[\mathbf{F}_{R} =\mathbf{W}_{R}\cdot\mathbf{R}+\mathbf{b}_{R} \tag{4}\] \[\mathbf{F}_{A} =\mathbf{W}_{A}\cdot\mathbf{A}+\mathbf{b}_{A}\]  
where \(\mathbf{W}_{R}\) and \(\mathbf{W}_{A}\) are parameter matrices for the relationship features \(\mathbf{F}_{R}\) and attribute features \(\mathbf{F}_{A}\), respectively. Graph Structure Embedding: To mimic the structural similarity between \(G_{s}\) and \(G_{t}\), capturing the proximity of entities and relationships, a GCN is employed to extract graph structural information. Specifically, a graph can be defined as \(G=(V,b)\), where \(V\) is a series of nodes \(\{\mathbf{v}_{1},\mathbf{v}_{2}\dots,\mathbf{v}_{G}\}\), and \(b\) represents the edge set. The entire feature matrix \(\mathbf{X}\in\mathbb{R}^{N\times G}\) comprises \(N\) feature vectors, \(\mathbf{X}=[\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{N}]^{T}\). The sparse symmetric adjacency matrix, denoted by \(\mathbf{A}\in\mathbb{R}^{N\times N}\), reflects the connection between each pair of nodes. \(\mathbf{A}_{ij}\) can be computed using the following radial basis function (RBF):  
\[\mathbf{A}_{ij}=\exp(-\frac{\left\|\mathbf{x}_{i}-\mathbf{x}_{j}\right\|^{2}}{\gamma_{1}}) \tag{5}\]where the parameter \(\gamma_{1}\) is empirically set to control the width of the RBF. The diagonal matrix is defined as \(\mathbf{D}=diag(d_{1},d_{2},\dots,d_{N})\), where \(d_{i}=\sum_{j=1}^{M}\mathbf{A}_{ij}\) represents the sum of the \(i\)-th row of the adjacency matrix. The multi-layer GCN at the \(l\)-th layer is represented as  
\[\mathbf{H}^{(l+1)}=[\widetilde{\mathbf{D}}^{-\frac{1}{2}}\widetilde{\mathbf{M}}\widetilde{ \mathbf{D}}^{-\frac{1}{2}}\mathbf{H}^{(l)}\mathbf{W}^{(l)}] \tag{6}\]  
where \([\cdot]_{+}\) represents the ReLU activation function, \(\widetilde{\mathbf{M}}=\mathbf{M}+\mathbf{I}_{N}\) is the adjacency matrix of \(G_{s}\bigcup G_{t}\) plus the identity matrix (self-connections), \(\widetilde{\mathbf{D}}\) is the trainable layer-specific weight matrix, \(\mathbf{H}^{(l)}\in\mathbb{R}^{N\times D}\) is the output of the previous layer of GCN, where \(N\) is the number of entities and \(D\) is the feature dimensionality. \(\mathbf{H}^{(0)}\) is randomly initialized, and the output of the last layer of GCN is used as the embedded graph structure \(\mathbf{F}_{G}\). Visual Embedding: Resnet-152 has been pre-trained on the ImageNet recognition task and serves as the feature extractor for all images. For each image, we use trainable Resnet-152 to extract image features and use the output of the last layer as a feature representation to obtain visual embedding:  
\[\mathbf{F}_{I}=\mathbf{W}_{I}\cdot\text{ResNet}(I)+\mathbf{b}_{I} \tag{7}\]  
The visual representations extracted by Resnet are expected to capture both low-level similarity and high-level semantic correlation between images.