# Big Data Integration in Remote Sensing across a Distributed Metadata-Based Spatial Infrastructure
## 3 Distributed Integration of Multi-Source Remote Sensing Data
### Distributed Multi-Source Remote Sensing Data Integration  
Distributed multi-source remote sensing data integration refers to the process of validating, inserting, updating, or deleting metadata in the main center metadata management system; it affects only the metadata for the distributed data providing sub-centers. The metadata management is mainly realized by the components of OODT, including the OODT crawler, OODT push-pull, and OODT file manager ([PERSON] et al., 2018) (see Figure 2).  
In the main data center, the push-pull daemon will be launched automatically by using its daemon launcher at the defined time interval. The daemon will wrap one of two processes: (1) RemoteCrawler, or (2) ListRetriever. The RemoteCrawler process crawls remote sites for files in the distributed sub-centers. Meanwhile, the RemoteCrawler process also automatically extracts metadata and transforms them into the ISO 19115-2:2009-based uniform metadata format. The ListRetriever retrieves known files from remote sites in the distributed sub-centers (that is, the path and file name to each file is known and has been specified in a property file, and a parser for that property file has been specified). After crawling or retrieval, the push-pull framework will be responsible for downloading remote content (pull), or accepting the delivery of remote content (push) to the main center for use by the LocalCrawler for ingestion into the file manager. Here, the remote content includes the metadata file and thumbnail of remote sensing data. It is worth mentioning that the LocalCrawler is developed in the main center, and is primarily responsible for crawling the local client system for files in the main center. The file manager component is responsible for tracking, ingesting, and moving metadata and thumbnails between a client system and a server system in the main center. Finally, the remote sensing metadata will be indexed by the SolrCloud, and their corresponding thumbnails will be archived in the file system.  
Both the RemoteCrawler and LocalCrawler have an incremental control mechanism in order to avoid duplicate data ingestion. In the intervals between crawling and data ingestion, the RemoteCrawler executes a Message Digest 5 (MD5) file verification process between the remote sites' files in the sub-center and the archived files in the main center. If the file has been archived in the main center, data ingestion will be stopped; otherwise, data ingestion continues. The LocalCrawler implements the second MD5 file verification process between the files in the client system (files from sub-centers downloaded to the main center) and the server system (archived files in the main center).  
Figure 2: The process of distributed data integration.  
If the files have been ingested and moved into the server system, the data ingestion will be stopped; otherwise, it continues.  
In addition, there is also the DaemonManager, in which the DaemonLauncher will register each daemon it creates. The DaemonManager ensures that no two Daemons are ever running at the same time. If a daemon is running when another requests permission to run, permission will be denied and the daemon will be added to the wait queue until the current running daemon and all other daemons ahead of it in the queue complete their tasks [20].