Package: prefect-shell
Integrating shell commands into your dataflow withprefect-shellVisit the full docshereto see additional examples and the API reference.The prefect-shell collection makes it easy to execute shell commands in your Prefect flows. Check out the examples below to get started!Getting StartedIntegrate with Prefect flowsWith prefect-shell, you can bring your trusty shell commands (and/or scripts) straight into the Prefect flow party, complete with awesome Prefect logging.No more separate logs, just seamless integration. Let's get the shell-abration started!fromprefectimportflowfromdatetimeimportdatetimefromprefect_shellimportShellOperation@flowdefdownload_data():today=datetime.today().strftime("%Y%m%d")# for short running operations, you can use the `run` method# which automatically manages the contextShellOperation(commands=["mkdir -p data","mkdir -p data/${today}"],env={"today":today}).run()# for long running operations, you can use a context managerwithShellOperation(commands=["curl -O https://masie_web.apps.nsidc.org/pub/DATASETS/NOAA/G02135/north/daily/data/N_seaice_extent_daily_v3.0.csv",],working_dir=f"data/{today}",)asdownload_csv_operation:# trigger runs the process in the backgrounddownload_csv_process=download_csv_operation.trigger()# then do other things here in the meantime, like download another file...# when you're ready, wait for the process to finishdownload_csv_process.wait_for_completion()# if you'd like to get the output lines, you can use the `fetch_result` methodoutput_lines=download_csv_process.fetch_result()download_data()Outputs:14:48:16.550|INFO|prefect.engine-Createdflowrun'tentacled-chachalaca'forflow'download-data'14:48:17.977|INFO|Flowrun'tentacled-chachalaca'-PID19360triggeredwith2commandsrunninginsidethe'.'directory.14:48:17.987|INFO|Flowrun'tentacled-chachalaca'-PID19360completedwithreturncode0.14:48:17.994|INFO|Flowrun'tentacled-chachalaca'-PID19363triggeredwith1commandsrunninginsidethePosixPath('data/20230201')directory.14:48:18.009|INFO|Flowrun'tentacled-chachalaca'-PID19363streamoutput:%Total%Received%XferdAverageSpeedTimeTimeTimeCurrentDl14:48:18.010|INFO|Flowrun'tentacled-chachalaca'-PID19363streamoutput:
oadUploadTotalSpentLeftSpeed00000000--:--:----:--:----:--:--014:48:18.840|INFO|Flowrun'tentacled-chachalaca'-PID19363streamoutput:111630k11192k00229k00:00:07--:--:--0:00:07231k14:48:19.839|INFO|Flowrun'tentacled-chachalaca'-PID19363streamoutput:831630k831368k00745k00:00:020:00:010:00:01747k14:48:19.993|INFO|Flowrun'tentacled-chachalaca'-PID19363streamoutput:1001630k1001630k00819k0014:48:19.994|INFO|Flowrun'tentacled-chachalaca'-PID19363streamoutput:
:00:010:00:01--:--:--821k14:48:19.996|INFO|Flowrun'tentacled-chachalaca'-PID19363completedwithreturncode0.14:48:19.998|INFO|Flowrun'tentacled-chachalaca'-Successfullyclosedallopenprocesses.14:48:20.203|INFO|Flowrun'tentacled-chachalaca'-FinishedinstateCompleted()!!! info "Utilize Previously Saved Blocks"You can save commands within a `ShellOperation` block, then reuse them across multiple flows, or even plain Python scripts.

Save the block with desired commands:

```python
from prefect_shell import ShellOperation

ping_op = ShellOperation(commands=["ping -t 1 prefect.io"])
ping_op.save("block-name")
```

Load the saved block:

```python
from prefect_shell import ShellOperation

ping_op = ShellOperation.load("block-name")
```

To [view and edit the blocks](https://orion-docs.prefect.io/ui/blocks/) on Prefect UI:

```bash
prefect block register -m prefect_shell
```ResourcesFor more tips on how to use tasks and flows in a Collection, check outUsing Collections!InstallationInstallprefect-shellwithpip:pipinstall-Uprefect-shellA list of available blocks inprefect-shelland their setup instructions can be foundhere.Requires an installation of Python 3.7+.We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.These tasks are designed to work with Prefect 2. For more information about how to use Prefect, please refer to thePrefect documentation.FeedbackIf you encounter any bugs while usingprefect-shell, feel free to open an issue in theprefect-shellrepository.If you have any questions or issues while usingprefect-shell, you can find help in either thePrefect Discourse forumor thePrefect Slack community.Feel free to star or watchprefect-shellfor updates too!ContributingIf you'd like to help contribute to fix an issue or add a feature toprefect-shell, pleasepropose changes through a pull request from a fork of the repository.Here are the steps:Fork the repositoryClone the forked repositoryInstall the repository and its dependencies:pip install -e ".[dev]"Make desired changesAdd testsInsert an entry toCHANGELOG.mdInstallpre-committo perform quality checks prior to commit:pre-commit installgit commit,git push, and create a pull request