# ReFuse: Generating Imperviousness Maps from Multi-Spectral Sentinel-2 Satellite Imagery
## 3 Materials and Methods
### Distributed Inference Pipeline  
Serving a model in production is crucial, especially when we need to serve a web-based application. Once the model is trained, we need to deploy it in a way that it can serve the application. Building highly available, scalable, distributed systems for machine-learning data pipelines is a complex task. In this work, we managed the whole process as a workflow, a sequence of tasks representing units of business logic. Figure 7 shows the workflow implemented for the prediction pipeline. The workflow is a Directed Cyclic Graph (DAG) where each node is a task and arrows are dependencies between tasks. In the extraction step, from the AoI and date range provided by the user, a first task searches and downloads Sentinel-2 data. Any images with clouds are appropriately removed from the process. The prediction is performed on the downloaded images after a subdivision into patches of the size required by the deep-learning model. Finally, merging predicted patches, the map is assembled and stored for later visualization. We implemented the whole approach through a Workflow Management System (WfMS) (Source code is available at [[https://github.com/priamus-lab/ReFuse](https://github.com/priamus-lab/ReFuse)]([https://github.com/priamus-lab/ReFuse](https://github.com/priamus-lab/ReFuse)) accessed on 22 September 2022). A WfMS provides an infrastructure for a defined workflow's setup, performance and monitoring. Therefore, involving a WfMS gives flexibility and extendibility to the approach because it implements off-the-shelf features such as data-sharing between tasks, recovery from failure, retrying failed tasks, task scheduling for batch runs, flow versioning and more. In this study, we use Prefect ([[https://www.prefect.io/accessed](https://www.prefect.io/accessed)]([https://www.prefect.io/accessed](https://www.prefect.io/accessed)) on 31 July 2022), an open-source orchestration and workflow tool. However, despite the choice made, this work aims to show a general approach, easily replicable with similar tools. We can highlight five essential layers in every WfMS (Figure 8): (i) the _orchestration layer_, which is responsible for the workflow's life cycle; (ii) the _agents_, daemon  
Figure 7: Schematic representation of the inference pipeline workflow. like processes that look for tasks and run them if any are available; (iii) the _flow run layer_, which is responsible for defining where the flow run; (iv) the _execution layer_, which manages where and how single tasks within the flow run; (v) the _storage layer_, which defines where the flow's code is stored, to be collected when needed. In addition, when tasks need to pass data between them, we need a _result layer_ which defines and manages where to store task results. It is worth pointing out that in a fully distributed system such as the one we are defining, the layers and components can be chosen from different types and deployed in different places. This aspect makes the solution extremely flexible: for example, in a development and test phase, one could choose to deploy an agent on a personal computer but let the flow run in the cloud. Figure 8 highlights in bold choices made in this work. Both agents and flows ran on a Kubernetes cluster; Prefect Cloud, a cloud-managed service, performed the orchestration. Flow's code and task results were stored in an object-based storage, in particular an AWS S3 bucket. Kubernetes is a system that manages containers where a container could be explained as a lightweight virtual machine. Containers encapsulate an application with all its dependencies, including system libraries, binaries, and configuration files, making it portable across different hosts. Kubernetes can create and scale these containerized applications automatically and manage storage among all the containers. Instead, object-based storage is a strategy that manages and manipulates data storage as distinct units called objects.