Package: prefect-planetary-computer
prefect-planetary-computerVisit the full docshereto see additional examples and the API reference.Prefect integrations with the MicrosoftPlanetary Computer(PC).OverviewThis collection includes aCredentials Block 🔑to store and retrieve a PC subscription key and Jupyter Hub token, with convenience methods to easily interact with the PCData Catalog 🌍andDask Gateway 🚀server.For more information about:using Azure services with Prefect and the Planetary Computer, check out theprefect-azurecollection.the integration between Prefect and Dask, check out theprefect-daskcollection.taking advantage of the Planetary Computer data catalog and compute resources, check out thePlanetary Computer documentation.ResourcesFor more tips on how to use tasks and flows in a Collection, check outUsing Collections!InstallationInstallprefect-planetary-computerwithpip:pipinstallprefect-planetary-computerRequires an installation of Python 3.8+.We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.These tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to thePrefect documentation.Usage!!! note
* The following Examples are adapted fromPlanetary Computer - Scale with Dask.- Require the following additional packages:
    ```
    pip install xarray zarr adlfs netcdf4 prefect_azure
    ```
- Make sure to share the same python dependencies - in particular `dask` and `distributed` - between the flow execution environment, the Dask Scheduler and Workers, [as explained in the Dask docs](https://docs.dask.org/en/stable/deployment-considerations.html#consistent-software-environments).Computing Dask CollectionsDask collection computations, such as Dask DataFrames, can be supported from within a Prefect task by creating a Dask Gateway cluster using the credentials block within the main flow or task itself.# Prefect tasks are executed using the default ConcurrentTaskRunner# Dask Collections tasks are executed on a new temporary Dask clusterimportxarrayasxrfromprefectimportflow,task,get_run_loggerfromprefect_planetary_computerimportPlanetaryComputerCredentialsfromprefect_azureimportAzureBlobStorageCredentialsfromprefect_azure.blob_storageimportblob_storage_uploadpc_credentials=PlanetaryComputerCredentials.load("PC_BLOCK_NAME")bs_credentials=AzureBlobStorageCredentials.load("BS_BLOCK_NAME")@taskdefcompute_mean(asset):logger=get_run_logger()withpc_credentials.new_gateway_cluster(name="test-cluster",image="pangeo/pangeo-notebook:latest")ascluster:cluster.adapt(minimum=2,maximum=10)client=cluster.get_client()ds=xr.open_zarr(asset.href,**asset.extra_fields["xarray:open_kwargs"],storage_options=asset.extra_fields["xarray:storage_options"])logger.info(f"Daymet dataset info\n:{ds}")timeseries=ds["tmin"].mean(dim=["x","y"]).compute()logger.info(f"Mean timeseries info\n:{timeseries}")returntimeseries@flowdefpc_dask_flow():# get a configured PySTAC clientcatalog=pc_credentials.get_stac_catalog()# compute the minimum daily temperature averaged over all of Hawaii,# using the Daymet datasetasset=catalog.get_collection("daymet-daily-hi").assets["zarr-abfs"]prefect_future=compute_mean.submit(asset)timeseries=prefect_future.result()# save NetCDF timeseries filetimeseries.to_netcdf("timeseries.nc")# upload to 'my-container' blob storage containerwithopen("timeseries.nc","rb")asf:blob=blob_storage_upload(data=f.read(),container="my-container",blob="timeseries.nc",blob_storage_credentials=bs_credentials,overwrite=False,)# return the blob name of the uploaded timeseries objectreturnblobpc_dask_flow()Using the Dask Task RunnerPrefect'sprefect_dask.DaskTaskRunnerautomatically instatiates a temporary Dask cluster at flow execution time, enabling submission of both Prefect and Dask Collections tasks.!!! warning
-prefect-daskrequires:distributed==2022.2.0; python_version < '3.8' distributed>=2022.5.0,<=2023.3.1- It requiresless configurationon the Dask Workers side when using Prefect Cloud, you canget startedfor free.# Both Prefect tasks and Dask Collections task are executed# on a new temporary Dask clusterimportxarrayasxrfromprefectimportflow,task,get_run_loggerfromprefect_planetary_computerimportPlanetaryComputerCredentialsfromprefect_azureimportAzureBlobStorageCredentialsfromprefect_azure.blob_storageimportblob_storage_uploadfromprefect_daskimportget_dask_clientpc_credentials=PlanetaryComputerCredentials.load("PC_BLOCK_NAME")bs_credentials=AzureBlobStorageCredentials.load("BS_BLOCK_NAME")pc_runner=pc_credentials.get_dask_task_runner(cluster_kwargs={"image":"pangeo/pangeo-notebook:latest",},adapt_kwargs={'minimum':1,'maximum':10,'active':True})@taskdefcompute_mean(asset):logger=get_run_logger()withget_dask_client()asclient:ds=xr.open_zarr(asset.hr**asset.extra_fields["xarray:open_kwargs"],storage_options=asset.extra_fields["xarray:storage_options"])logger.info(f"Daymet dataset info\n:{ds}")timeseries=ds["tmin"].mean(dim=["x","y"]).compute()logger.info(f"Mean timeseries info\n:{timeseries}")returntimeseries@flow(task_runner=pc_runner)defpc_dask_flow():# get a configured PySTAC clientcatalog=pc_credentials.get_stac_catalog()# compute the minimum daily temperature averaged over all of Hawaii,# using the Daymet datasetasset=catalog.get_collection("daymet-daily-hi").assets["zarr-abfs"]mean_task=compute_mean.submit(asset)timeseries=mean_task.result()# save NetCDF timeseries filetimeseries.to_netcdf("timeseries.nc")# upload to 'my-container' blob storage containerwithopen("timeseries.nc","rb")asf:blob=blob_storage_upload(data=f.read(),container="my-container",blob="timeseries.nc",blob_storage_credentials=bs_credentials,overwrite=False,)# return the blob name of the uploaded timeseries objectreturnblobpc_dask_flow()FeedbackIf you encounter any bugs while usingprefect-planetary-computer, feel free to open an issue in theprefect-planetary-computerrepository.If you have any questions or issues while usingprefect-planetary-computer, you can find help in either thePrefect Discourse forumor thePrefect Slack community.Feel free to star or watchprefect-planetary-computerfor updates too!ContributingIf you'd like to help contribute to fix an issue or add a feature toprefect-planetary-computer, pleasepropose changes through a pull request from a fork of the repository.Here are the steps:Fork the repositoryClone the forked repositoryInstall the repository and its dependencies:pip install -e ".[dev]"Make desired changesAdd testsInsert an entry toCHANGELOG.mdInstallpre-committo perform quality checks prior to commit:pre-commit installgit commit,git push, and create a pull request