# Big Data Integration in Remote Sensing across a Distributed Metadata-Based Spatial Infrastructure
## 1 Introduction  
Since Landsat-1 first started to deliver volumes of pixels in 1972, the amount of archived remote sensing data stored by data centers has increased continuously [1, 2]. According to incomplete statistics, the total amount of data archived by the Earth Observing System Data and Information System (EOSDIS) reached 12.1 petabytes (PBs) around the year 2015 [3]. Up until August 2017, the archived data volume of China National Satellite Meteorological Center (NSMC) reached 4.126 PBs [4], and the China Center for Resources Satellite Data and Application (CCRSDA) archived more than 16 millionscenes of remote sensing images [5; 6]. Such large amounts of remote sensing data have brought great difficulties in terms of data integration and management of each data center. For data integration, due to various satellite orbit parameters and the specifications of different sensors, the storage formats, projections, spatial resolutions, and revisit periods of the archived data are vastly different, and these differences have resulted in great difficulties for data integration. In addition, the remote sensing data received by each data center arrives continuously at an ever-faster code rate. It is preferable to ingest and archive the newly received data in order to provide users with the latest data retrieval and distribution service [7]. Therefore, a unified metadata format and a well designed data integration framework are urgently needed. For data management, the massive amounts of remote sensing data mainly result in difficulties for data retrieval, but the retrieval efficiency is mainly dependent upon the data organization model and storage system. Currently, the two most widely used data organization models are: (1) spatio-temporal recording system-based satellite orbit stripes or scene organization; and (2) globally meshed grid-based data tiling organization [8]. However, the former has obvious shortcomings for massive data retrieval and quick access; and the latter causes an increase by about one-third in the amount of data due to image segmentation, thus requiring larger data storage spaces. Hence, we should explore a reasonable spatial organization mode for massive, multi-source remote sensing data, so as to improve the efficiency of massive data retrieval. As for the data storage system, the most widely used mode is the combination of the file system (FS) and database management system (DBMS). The \"DBMS-FS mixed management mode\" solves both the problems of metadata management and quick retrieval, and also maintains the high read/write efficiency of a file system. Therefore, in this paper, for data integration across a distributed data center spatial infrastructure, we firstly proposed an International Standardization Organization (ISO) 19115-based metadata transform method, and then adopted the internationally popular data system framework object-oriented data technology (OODT) [9] to complete the distributed remote sensing data integration. For data management, we proposed a logical segmentation indexing (LSI) model to achieve the organization of integrated remote sensing metadata, and chose SolrCloud to realize the distributed index and quick retrieval [10]. LSI model takes the logical segmentation indexing code as the identifier of each remote sensing data, rather than performing an actual physical subdivision. This not only increases the efficiency of data retrieval with the help of the global subdivision index, but also avoids generating numerous small files caused by the physical subdivision of data.