## 3 SAT-Hadoop-Processor
### RS Data Ingestion  
The processing chain of the RSBD takes in many challenges. First of all, satellite data are diffused into ground stations, so the big complaint is how to gather these data in NRT to keep data fresh. These data should be pre-processed to remove erroneous, inaccurate, and unneeded datasets to retain only data of interest and integrate them into a distributed and scalable storage platform. Figure 2 displays the six phases of the ingestion layer. The acquisition is the initial step in the satellite data processing. We acquired data automatically from the sources mentioned in Section 3.1. The downloaded files were compressed from the ground station and datacenters. Therefore, the following phase decompresses satellite data automatically by a Bash script (Tar, Zip, and bz2); thus, the number of files grew 240 times, and the size increased up to 40%. Based on these results, we confirm that RS data require more storage space and become more complex for processing after the decompression step. Commonly, the HDF5, NetCDF, BUFR, and BIN file formats are dedicated to storing RS data. These data require conversion from the scientific file format to a CSV or the Extensible Markup Language (XML) file format. We employed two Python libraries: the BUFRextract (BUFREXC) and the pybuff_ecmwf (ECMWF). Afterward, datasets are prepared to be extracted. So, the total size of data stays roughly the same after the conversion. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Data Source** & **CMD** & **Download Link** \\ \hline MDEO & CMD 1 & [[https://www.eumestast.int/eumectcast](https://www.eumestast.int/eumectcast)]([https://www.eumestast.int/eumectcast](https://www.eumestast.int/eumectcast)) \\ EOSDIS & CMD 1 & [[https://earthdata.nasa.gov/earth-observation-data](https://earthdata.nasa.gov/earth-observation-data)]([https://earthdata.nasa.gov/earth-observation-data](https://earthdata.nasa.gov/earth-observation-data)) \\ NESDIS & CMD 1 & [[ftp://ftp-npp.bou.class.noaa.gov](ftp://ftp-npp.bou.class.noaa.gov)]([ftp://ftp-npp.bou.class.noaa.gov](ftp://ftp-npp.bou.class.noaa.gov)) \\ COAH & CMD 2 \& 3 & [https://s5 phub.copernicus.eu](https://s5 phub.copernicus.eu) \\ MGS Madrid & CMD 1 & [[https://datos.madrid.es/portal/site/egob](https://datos.madrid.es/portal/site/egob)]([https://datos.madrid.es/portal/site/egob](https://datos.madrid.es/portal/site/egob)) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: RS data sources’ URLs. Figure 2: The data ingestion phases [3]. The downloaded data come from polar satellites flying in a Low Earth Orbit (LEO) with an altitude of 800 km and making 16 orbits daily. Thus, the processing of all data of the Earth takes more computing resources and a long execution time. We coded Python script filtering satellite data by countries using the longitude and latitude. We found that big countries, such as the USA, China, and Australia, have many files, reaching more than 700 files per day. However, the smallest state, which is Qatar, covers only about 50 files, and the data size is megabit (MB). The next step is data extraction. It permits the selection of the looked-for variables. For example, we were interested in 12 variables: temperature, humidity, pressure, wind speed, AOD, the Vertical Column Density (VCD) of trace gases, etc. The final step is the data integration into the HDFS, HBase, and Hive storage framework of Hadoop using some CMDs detailed in the Supplementary Materials File (SMF). The main algorithm in which the SAT-Hadoop-Processor was developed is as follows:  
ST is the system set of satellites of interest {st1,..., stn}. SS is the used satellite sensor {ss1,..., ssn}. P is the used satellite products {p1,..., pn}. F is the acquired satellite files {f1,..., Fn}. UF' is the unzipped RS files {uf'1,..., uf'n}. CF' is the converted RS files {cf'1,..., cf'n}. FF' is the orbit filtered RS files {ff'1,..., ff'n}. EF' is the extracted RS files {ef'1,..., ef'n}. V is the used satellite variables {v1,..., vn}. D is the extracted RS dataset {d1,..., dn}.