# Multi-Modal Spatio-Temporal Knowledge Graph of Ship Management
## 3 Method
### Multi-Modal Spatio-Temporal Ontology Construction
#### 3.1.1 Preliminary Preparation  
Analyzing and determining the domain and scope of the ontology to be constructed from the existing data and the purpose of using ontology plays an important role. Not only can this step ensure that the designed ontology meets the practical needs of the application, but it is of great significance to the development and maintenance of the ontology. There are \"platform (_Plat_)\", \"sensor (_Sensor_)\", \"communication equipment (_Equipment_)\", \"event (_Event_)\", \"weather (_Wather_)\", \"area (_Area_)\", and other relevant ship communication data from MySQL databases. Several types of data information, such as _Event_ and _Area_, involve time and spatial information. For example, _Area_ includes such spatial information as longitude, latitude, and height. Furthermore, _Sensor_ and _Equipment_ will generate multi-modal data, such as images and videos during ship communication. These data are of great significance to the analysis and prediction of ship communication situations. Therefore, it can be determined that the ontology constructed in this paper needs to achieve unified constraints and correlation integration of multi-source heterogeneous, spatio-temporal multi-modal data in ship communication, and further provide outstanding data management support for subsequent ship communication situation analysis and prediction tasks. There are three methods to reuse existing ontologies for improving the quality and efficiency of ontology construction: (1) extending existing ontologies, (2) reusing existing ontologies, and (3) integrating multiple existing ontologies. In this paper, we design a multi-modal spatio-temporal ontology based on ship communication by integrating multiple existing ontologies. Specifically, multi-modal spatio-temporal ontology is specifically designed to integrate and describe time and spatial information. There is an existing Time Ontology in OWL [15] that provides a clear, formal, and standardized description of time concepts and the relations between them. Therefore, we incorporate concepts _Instant_ and _Date-TimeDescription_ from the Time Ontology in OWL to define the time-related description in our ontology. A set of geospatial data types, functions, and predicates have been defined in the existing ontology-based query language extension GeoSPARQL [16] for processing geospatial data. We can abstract geospatial concepts and relations between concepts from GeoSPARQL to form the geospatial ontology. For example, _Coordinate Reference System (CRS)_ is used to determine the position and shape of geospatial data. Additionally, we reuse the structure of existing event ontology to define the ship communication event description and its related semantic information. Table 1 illustrates the reusing concepts and their corresponding descriptions. The construction of a multi-modal spatio-temporal ontology involves extracting important terms from the data and analyzing their context to accurately identify their meanings and relations. This process is crucial for guiding the core structure design and semantic relations construction of the ontology. As the ontology to be constructed is a multi-modal spatio-temporal ontology, it is necessary to extract professional terms and spatio-temporal information-related terms from ship communication data, with a specific focus on extracting important terms from multi-modal data. The MySQL databases contain multi-modal information, including text, images, and videos. Therefore, this paper considers the application of various technical methods to extract high-quality important terms for constructing a comprehensive and accurate multi-modal spatio-temporal ontology, taking into account both the structural information of the MySQL databases and the characteristics of the multi-modal data. Specifically, various techniques, such as natural language processing, image processing, and video analysis can be employed to extract relevant terms from the data.