# Mass Processing of Sentinel-1 Images for
## 3 Methods
### Platform for Earth Observation Data Storage and Processing: the JEODPP  
With its fleet of Sentinel satellites operated by the European Space Agency, the Copernicus programme of the European Union is making Earth Observation truly enter the big data era. Indeed, the Sentinel satellite series in full operational capacity will produce a continuous and voluminous stream of image data (up to 10 terabytes per day) with a variety of sensors at different spectral, spatial, and temporal resolutions. The resulting vast amount of free and open data generated by the Sentinel and other Earth Observation satellites calls for new approaches in data management and processing to enable the timely extraction of relevant information from these data in combination with data from other sources. Indeed, a fragmented approach whereby individual projects are managing their data and processing infrastructure is not sustainable anymore. At the Joint Research Centre, this motivated the development of a common platform to serve the needs of projects relying on the processing and analysis of Earth Observation data in support of policy needs. This platform is called the JRC Earth Observation Data and Processing Platform (JEODPP) [29; 30]. It is versatile in the sense that it accommodates existing scientific workflows such as those based on the SUMO software without the need for rewriting the underlying code to match specific requirements. The main components of the JEODPP are briefly detailed hereafter with emphasis on those used for the massive processing of Sentinel-1 imagery for maritime surveillance. The JEODPP relies on commodity hardware for both storage and processing components as well as a series of software layers enabling different access levels to users. It is scalable, but at the time of this study, the storage component consisted of 1.4 petabyte organised in 10 storage servers each equipped with 24 disks of 6 terabytes. This type of storage architecture is referred to as Just a Bunch of Disks (JBODs). The whole disk space is seen as a unique logical volume thanks to the distributed file system developed by CERN and called EOS [31; 32]. To secure high availability and the correction of errors, all data are stored with a redundancy level equal to 2 leading to a net storage capacity of 0.7 petabyte. The processing component consists of 472 processors originating from 23 servers (16 with 12 cores and 7 with 40 cores). The data required for this study were discovered and downloaded automatically using OpenSearch and OData scripting capabilities offered by the Copernicus Open Access [33] and Copernicus Services [34] hubs operated by the European Space Agency. The query for download consisted of searching for all Sentinel-1A IW-GRDH products with a non-empty intersection with the Mediterranean Sea during the period 3 October 2014-30 September 2016 as detailed in Section 2.2. The downloaded files are unzipped during ingestion for subsequent use by SUMO. The analysis of the revisit times with the joint Sentinel-1A + Sentinel-1B (Section 4.1) was done by performing a similar query to that described above but retrieving only the necessary metadata files. To accommodate a wide variety of legacy software written in different programming languages with often incompatible software library dependencies, the JEODPP implements virtualisation technologies. The typical overhead encountered with operating system virtualisation is avoided thanks to operating system level virtualisation (also called lightweight virtualisation) based on Docker containerization [35]. Each application runs within a Docker container isolating the user-space instance to prevent conflict with applications running side-by-side in different containers. A Docker container is launched from a Docker image that contains all the required software and associated libraries. The SUMO software relies on programs written in Java so it was enough to create a Docker image equipped with Java run time environment.