# Multi-Modal Spatio-Temporal Knowledge Graph of Ship Management
## 3 Method
### Multi-Modal Spatio-Temporal Knowledge Graph Generation
#### 3.2.1 Knowledge Extraction from Unstructured Data
Finally, the predicted results of each token are post-processed to achieve the entire named entity recognition task. For the speech modality, its processing can be briefly summarized as mapping the raw speech signal into a continuous space. Specifically, the feature sequence of the speech is transformed into the corresponding character sequence for subsequent operations and calculations. Since the speech sequence can be described as a two-dimensional spectrogram with a time axis and a frequency axis, its feature sequence is usually several times longer than the character sequence. When reading spectrograms, humans rely on the correlation between different frequencies over time to predict pronunciation. Therefore, focusing on the time and frequency axes may be advantageous for modeling the temporal and spectral dynamics in the spectrogram. We choose convolutional neural networks to exploit the structural locality of the spectrogram and alleviate length mismatch across time, ultimately transforming it into an input sequence that can be accepted by the Transformer encoder. **Encoder**  
In the encoder part, we employed three different Transformer encoders to fully extract the feature information of each modality, namely text, image, and speech. The adoption of distinct Transformer encoders is based on the unique features of each modality. Using different encoders can train the model parameters to better fit the needs of each modality. However, these three Transformer encoders all retain the characteristics of Transformer, such as dot-product attention and multi-head attention mechanism. The Transformer encoder for text and speech modalities retains the position-wise feedforward neural network, while the Transformer encoder for the image modality uses a multilayer perceptron and employs GeLU as its activation function, abandoning the traditional ReLU activation function in traditional Transformers. **Decoder**  
To better accomplish the tasks of entity recognition and relation extraction in semantic extraction, we used different decoders based on the Transformer encoder and employed different linear layers in the final hidden output state to perform the tasks. For enhanced entity recognition, we incorporated a conditional random field (CRF) decoder in the decoder part. This allows for the better utilization of dependencies among different labels. For a given feature sequence \(s=[s_{1},s_{2},\dots,s_{T}]\) and its corresponding gold label sequence \(y=[y_{1},y_{2},\dots,y_{T}]\), where \(Y(s)\) represents the  
where \(f(y_{t-1},y_{t},s)\) calculates the transition score from \(y_{t-1}\) to \(y_{t}\) and the score of \(y_{t}\). The optimization goal is to maximize \(P(y|s)\), and the Viterbi algorithm is used to find the path with the maximum probability during decoding. In relation extraction, the identification of relations can be viewed as a classification problem. Entity relation types are generally mutually exclusive, although there are a few non-mutually exclusive relations, which account for a low proportion and can be artificially decomposed into mutually exclusive relations. Since Softmax is well-suited for handling mutually exclusive multi-classification problems, a Softmax classifier is employed to classify the output generated by the Transformer encoding layer.