# Parallel Processing Strategies for Geospatial Data in a Cloud Computing Infrastructure
## 4 Abstracting Platform-Specific Dependencies with openEO
The two most time-consuming processes of the workflow were: creating the data cube and spatially aggregating the pixels within the polygons. They accounted for 97% of the total elapsed processing time. Other processes included: querying the Sentinel-2 data catalog, masking cloudy pixels, reading the parcel data, calculating the NDVI, and writing the results. To create the data cube, all 71 images that cover the region of interest had to be read from the disk. For a single-core and no tiling applied, the processing time to create the data cube was 986 s (55% of the total time). The time for the spatial aggregation was 755 s (42%). Speedup by tiling was assessed first (see Figure 4). Comparing the experimental speedup with the theoretical speedup using [PERSON]'s law, the parallelization was estimated as \(P=95\%\) (dashed curve in Figure 4). The speedup by multi-threading was then measured by comparing the actual elapsed processing time as a function of the number of cores. Speedup values were calculated for the two most time-consuming processes individually and for the overall process. As shown in Figure 5, the speedup obtained via multi-threading varied with the tiling scheme. Whereas more and smaller tiles provide better parallelization through tiling, multi-threading performs better for fewer and larger tiles. This can be explained by the relative increase in the processing time of the serial tasks when dealing with smaller tiles. The effect is much more pronounced for the process to create the data cube (Figure 5a) due to the relative importance of the overhead involved with reading only small parts of the image, such as: opening and closing a file, extracting the compressed image in JP2 format, and reading the metadata. Figure 4: Experimental speedup via tiling (solid line) with theoretical speedup corresponding to [PERSON]’s law (parallelization \(P=95\%\), dashed line). The process to create the data cube has a relatively low level of parallelization. Similar to the spatial tiling, the parallelization level was estimated by comparing the experimental speedup obtained with multi-threading with the theoretical speedup based on [PERSON]'s law [11]. Figure 5: Experimental (solid) and (theoretical) speedup via multi-threading for different processes and tiling schemes: creating data cube (**a**), spatial aggregation (**b**), and all processes combined (**c**). Speedup increases for larger and fewer tiles. In the case of creating the data cube, the level of parallelization was estimated between \(P=77\%\) (no tiling) and \(P=40\%\) (81 tiles). The process involved with spatial aggregation achieved the highest level (Figure 5b): between \(P=95\%\) (no tiling) and \(P=73\%\) (81 tiles). For all processes in the workflow using multi-threading, \(P\) ranged from 83% (no tiling) to 42% (81 tiles). Tiling and multi-threading were then combined. The objective here was to select the tiling scheme and number of threads that optimized the total speedup in a typical cloud processing environment. The number of processing cores and the amount of memory available to a specific user and workflow must hereby be taken into account. In practice, this depends on the number of cores available per working node, the applied policy with respect to concurrent users and workflows, and the costs involved with cloud services. Speedup was first tested by processing multiple tiles single-threaded in parallel. With a tiling scheme of 81 tiles, a speedup of 21.3 was obtained (lower curve in gray in Figure 6). Speedup was further increased by combining tiling with multi-threading. The results are shown in Figure 6 in blue (4 threads) and red (30 threads). The maximum speedup was reached for 36 tiles. This corresponded to tile sizes of 600 \(\times\) 600 pixels. Further tiling and adding more threads did not improve speedup. This can be explained by the following effects. As shown in Figure 5, the speedup with multi-threading is reduced when the tile size decreases.