# Multi-Modal Spatio-Temporal Knowledge Graph of Ship Management
## 3 Method
### Multi-Modal Spatio-Temporal Ontology Construction
#### 3.1.1 Preliminary Preparation
The extracted terms can then be utilized to construct a comprehensive and accurate ontology that effectively represents the spatio-temporal information. **Extract terms from structure information of data**  
The data type and storage structure related to equipment information in the MySQL database are presented in Table 2. To extract important terms, we analyze and parse the structure and content information of the relational data table in the MySQL databases. The table name \"communication equipment (Equipment)\" is identified as an important concept term, as well as column names, such as \"identifier (id)\", \"name\", \"nation\", \"type\", \"model\", \"description\", \"image\", and \"status\" are also recognized as terms. For the content information in the table, we employ data preprocessing (including data cleaning and segmentation) and text mining techniques such as term frequency statistics and TF-IDF algorithms to automatically extract important terms. For example, shortwave communication equipment terms such as \"7300 type\" and \"726 shortwave communication equipment\" are extracted from the \"model\" attribute. **Extract terms from multi-modal data**  
Ship communication involves multimedia modal data, including images, audio, and video. It is crucial to establish comprehensive correlations among the different modalities while integrating them. * For image data in ship communication, both image recognition and manual annotation technology can be comprehensively applied to identify frequent image regions and use their labels as terms, where image recognition technology includes image preprocessing such as image denoising and image enhancement, as well as image segmentation and object detection techniques. Manual annotation technology, on the other hand, is employed to annotate objects and scenes in images, facilitating the extraction of relevant terms. * For audio data in ship communication, the extraction of terms can be accomplished using audio analysis and manual annotation technology. Similar to extracting terms from images, we can annotate objects and scenes in audio data to obtain relevant terms. After automatic speech recognition and speech-to-text conversion, audio classification \begin{table}
\begin{tabular}{c c} \hline \hline Name & Description \\ \hline Instant & Representing the occurrence time of a communication event. \\ \hline DateTimeDescription & Describing time information including year, month, day, hour, minute, second, etc. \\ \hline CRS & Indicating coordinate reference systems (e.g., \"WGS84\", \"UTM,\" etc.) for determining the position and shape of geospatial data. \\ \hline Event & Linking dynamic ship communication information. \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Reusing concepts and corresponding descriptions. \begin{table}
\begin{tabular}{c c} \hline \hline Communication Equipment(Equipment)( \\ id INT PRIMARY KEY AUTO\_INCREMENT, \\ name VARCHAR(64) NOT NULL, \\ nation VARCHAR(64) NOT NULL, \\ type VARCHAR(64) NOT NULL, \\ model VARCHAR(64) NOT NULL, \\ description TEXT NOT NULL, \\ image BLOB, \\ status VARCHAR(64) NOT NULL, \\ ) & - Equipment malfunction situation \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Equipment data storage form in MySQL database. and active speech detection techniques can be used to detect frequent entity targets from audio data and define their labels as terms. * For video data in ship communication, image recognition can be employed to identify frequent image regions and assign their labels as terms after video preprocessing operations including video frame segmentation, inter-frame difference, image denoising, and image enhancement. Additionally, manual annotation technology can be utilized to annotate objects and scenes in the video, enabling the extraction of relevant terms.