Package: eo-grow
eo-growEarth observation framework for scaled-up processing in Python.Analyzing Earth Observation (EO) data is complex and solutions often require custom tailored algorithms. In the EO domain most problems come with an additional challenge:How do we apply the solution on a larger scale?Working with EO data is made easy by theeo-learnpackage, while theeo-growpackage takes care of running the solutions at a large scale. Ineo-growanEOWorkflowbased solution is wrapped in apipelineobject, which takes care of parametrization, logging, storage, multi-processing, EOPatch management and more. However,pipelinesare not necessarily bound toEOWorkflowexecution and can be used for other tasks such as training ML models.Features ofeo-growinclude:Direct use ofEOWorkflowprocedures.Parametrizing workflows by using validated configuration files, making executions easy to reproduce and adjust.Easy use of both local and AWS S3 storage with no required code adaptation.Splitting large areas of interest into grids and defining collections of EOPatches.Workflows can be run either single-process, multi-process, or even on multiple machines (by usingrayclusters).Execution reports and customizable logging.Options for skipping already processed data when re-running a pipeline.Offers a CLI interface for running pipelines, validating configuration files, and generating templates.A collection of basic pipelines, with methods that can be overridden to tailor to a large amount of use-cases.General Structure OverviewThe core object ofeo-growis thePipeline. Each pipeline has arun_proceduremethod, which is executed after the pipeline is set up. By default, therun_procedureexecutes anEOWorkflowwhich is built by the (user-defined)build_workflowmethod.Each pipeline is linked to so calledmanagers:StorageManagerhandles loading and saving of files,AreaManagerdefines the area of interest and how it should be split into EOPatches,LoggingManagerprovides control over logging.Managers and pipelines usually require a large amount of parameters (setting storage paths, configuring log parameters, etc.), which are provided in.jsonconfiguration files. Eacheo-growobject contains a specialSchemaclass, which is apydanticmodel describing the parameters of the object. Config files are then validated before execution to catch issues early. Templates for config files can be generated with theeogrow-templateCLI command.To make config files easier to writeeo-growuses a simpleconfig languagethat supports importing other configs, variables, and more.InstallPyPi distributionTheeo-growpackage requires Python version>= 3.8and can be installed withpip install eo-growCommand Line InterfaceRunning pipelines is easiest by using the CLI provided byeo-grow. For all options use the--helpflag with each command.eogrow <config>executes the pipeline defined in the<config>file.eogrow-validate <config>only performs validation of the<config>file.eogrow-test <config>initializes the pipeline/object but does not run it. Useful for testing if managers are set correctly or for generating area-split grids.eogrow-ray <cluster> <config>executes the pipeline defined in<config>on the active Ray cluster defined by the<cluster>file.eogrow-template <import path> <template>generates a template config for the object specified by the<import path>and saves it to the<template>file (or outputs it directly if<template>is not provided).DocumentationFor more information on the package visitreadthedocs.Explanatory examples can be foundhere.More details on the config language used byeo-growcan be foundhere.Questions and IssuesFeel free to ask questions about the package and its use cases atSentinel Hub forumor raise an issue onGitHub.LicenseSeeLICENSE.AcknowledgementsThis project has received funding from the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 101004112.