# Multi-Modal Spatio-Temporal Knowledge Graph of Ship Management
## 3 Method
### Multi-Modal Spatio-Temporal Knowledge Graph Generation
#### 3.2.1 Knowledge Extraction from Unstructured Data  
This paper employs the Transformer technology [18], known for its exceptional performance in feature extraction tasks, to accomplish the semantic extraction task of multi-modal unstructured data, encompassing text, image, and speech modalities. Initially, the raw textual, image, and speech data are preprocessed and transformed into formats compatible with the Transformer model. Then, the processed input sequences are fed into the Transformer encoder through the encoder and decoder of the Transformer, and the features of each modality data are extracted through the internal multi-head attention mechanism and other modules of the encoder. Following encoding, the input sequence proceeds to the decoder, which is followed by specific output heads for multi-modal data and is used to output the extracted triplets from each modality, providing data support for constructing a multi-modal knowledge graph. **Data Preprocessing**  
Due to the different characteristics of data in different modalities, it is necessary to preprocess the data of each modality and convert it into an input format that can be accepted by the Transformer encoder. The subsequent section will provide an overview of the data preprocessing steps for each modality. For an image, its storage format in a computer is composed of individual pixels. Typically, each pixel of an image (assuming it is single-channel) is treated as a token, and its corresponding embedding operation is performed. Then, the embedding result is added to the corresponding positional encoding to obtain the final image embedding. However, for a single-channel image with a size of \(224\times 224\), treating each pixel as a token would result in an input length of 50,176 for the Transformer, which is too large and leads to an excessively large number of model parameters, making the model cumbersome and requiring more computational resources and time during training. To address this issue, we adopt a patch-based method, which involves dividing the original image into small patches and treating each patch as a token, as shown in Figure 7. For a three-channel color image, its size format is \([224,224,3]\). The size of each patch is set to \(16\times 16=256\). Therefore, the original image can be divided into \((224/16)^{2}=196\) patches. Since it is a three-channel image, the size of each patch is \(256\times 3=768\). Hence, after patch processing, a \(224\times 224\times 3\) image can be transformed into tokens with a size of \(196\times 768\), where num_token = 196 and token_dim = 768, which is in line with the input format required by Transformer. Before inputting it into the Transformer encoder, positional embedding needs to be added. After the aforementioned two steps of processing, the data can be inputted into the Transformer encoder for further processing. Figure 7: Patch-based image preprocessing. The processing of textual data is relatively straightforward. Firstly, the raw corpus is formatted into sentences. Then, each word in each processed sentence is embedded to obtain an embedding sequence. This sequence consists of multiple embedding tokens, where each token represents the embedding of a word. Afterward, a [CLS] token and a [SEP] token are added to the beginning and end of the embedding sequence, respectively. Before inputting the sequence into the model, padding processing is performed and a corresponding padding mask vector is constructed. The purpose of padding processing is to maintain a consistent length of input sentences. Since the length of text varies, Pad tokens are added to shorter texts to make the sentence lengths consistent, which facilitates subsequent model processing and computation. The processed embedding sequence, positional embedding, and semantic embedding are integrated to form the final embedding input vector. The embedding input vector is then inputted into the model for feature extraction, and a classification layer is used to classify each output token.