Saravanan666/pypi-org huggingface dataset:
Package: prefect
PrefectPrefect is an orchestration and observability platform for building, observing, and triaging workflows.
It's the simplest way to transform Python code into an interactive workflow application.Prefect allows you to expose your workflows through an API so teams dependent on you can programmatically access your pipelines, business logic, and more.
Prefect also allows you to standardize workflow development and deployment across your organization.With Prefect, you can build resilient, dynamic workflows that react to the world around them and recover from unexpected changes.
With just a few decorators, Prefect supercharges your code with features like automatic retries, distributed execution, scheduling, caching, and much more.Every activity is tracked and can be monitored with a self-hostedPrefect serverinstance or managedPrefect Clouddashboard.Getting startedPrefect requires Python 3.8 or later. Toinstall Prefect, run the following command:pipinstallprefectThen create and run a Python file that uses Prefectflowandtaskdecorators to orchestrate and observe your workflow that fetches the number of GitHub stars from a repository.fromprefectimportflow,taskfromtypingimportListimporthttpx@task(log_prints=True)defget_stars(repo:str):url=f"https://api.github.com/repos/{repo}"count=httpx.get(url).json()["stargazers_count"]print(f"{repo}has{count}stars!")@flow(name="GitHub Stars")defgithub_stars(repos:List[str]):forrepoinrepos:get_stars(repo)# run the flow!if__name__=="__main__":github_stars(["PrefectHQ/Prefect"])Fire up the Prefect UI to see what happened:prefectserverstartTo run your workflow on a schedule, turn it into a deployment and schedule it to run every minute by changing the last line of your script to the following:github_stars.serve(name="first-deployment",cron="* * * * *")You now have a server running locally that is looking for scheduled deployments!
Additionally you can run your workflow manually from the UI or CLI - and if you're using Prefect Cloud, you can even run deployments in response toevents.Prefect CloudStop worrying about your workflows.
Prefect Cloud allows you to centrally deploy, monitor, and manage the data workflows you support. With managed orchestration, automations, and webhooks, all backed by enterprise-class security, build production-ready code quickly and reliably.Read more about Prefect Cloudhereor sign up totry it for yourself.Next stepsThere's lots more you can do to orchestrate and observe your workflows with Prefect!
Start with ourfriendly tutorialor explore thecore concepts of Prefect workflows.Join the communityPrefect is made possible by the fastest growing community of thousands of friendly data engineers. Join us in building a new kind of workflow system. ThePrefect Slack communityis a fantastic place to learn more about Prefect, ask questions, or get help with workflow design. All community forums, including code contributions, issue discussions, and slack messages are subject to ourCode of Conduct.ContributeSee ourdocumentation on contributing to Prefect.Thanks for being part of the mission to build a new kind of workflow system and, of course,happy engineering!
Package: prefect-2-discord
prefect-discord-->Welcome!A collection containing a notification block making it easy to send messages and notifications from Prefect >= 2.4.1 to a Discord channel. Read on for instructions on how to use theDiscordWebhookblock.Getting StartedPython setupRequires an installation of Python 3.7+.We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.These tasks are designed to work with Prefect 2. For more information about how to use Prefect, please refer to thePrefect documentation.InstallationInstallprefect-discordwithpip:pipinstallgit+https://github.com/rpeden/prefect-discord.gitThen, register toview the blockon Prefect Cloud:prefectblockregister-mprefect_discordNote, to use theloadmethod on Blocks, you must already have a block documentsaved through codeorsaved through the UI.Creating a Discord Webhook BlockBefore you start, you'll need to add a webhook to one of your Discord channels by followingthese instructions. When you're done, copy the webhook URL.Via the UIClick on theBlocksmenu, then click the+button in the page header to open the block catalog:Find the listing for the Discord Webhook block and clickAdd:Add a name for your block, paste the Discord webhook URL you copied, and clickCreate:You'll see your new block's page:And your block is ready to use!In Pythonfromprefect_discordimportDiscordWebhookmy_webhook=DiscordWebhook(url="https://my-discord-webhook-url")my_webhook.save("awesome-discord-webhook")Use Discord Webhook Notificationsfromprefectimportflowfromprefect_discordimportDiscordWebhook@flowdefexample_flow():discord_webhook_block=DiscordWebhook.load("awesome-discord-webhook")discord_webhook_block.notify("Hello from Prefect!")example_flow()When you run this code, you should see this in your Discord channel:ResourcesIf you encounter any bugs while usingprefect-discord, feel free to open an issue in theprefect-discordrepository.If you have any questions or issues while usingprefect-discord, you can find help in either thePrefect Discourse forumor thePrefect Slack community.Feel free to ⭐️ or watchprefect-discordfor updates too!DevelopmentIf you'd like to install a version ofprefect-discordfor development, clone the repository and perform an editable install withpip:gitclonehttps://github.com/rpeden/prefect-discord.gitcdprefect-discord/

pipinstall-e".[dev]"# Install linting pre-commit hookspre-commitinstall
Package: prefect-alert
prefect-alertWelcome!Prefectis an open-source library that allows you to orchestrate and observe your dataflow defined in Python.prefect-alert is a decorator that allows you to send alert when a Prefect flow fails.Readthis articleif you are not familiar with how to send notifications with Prefect.Getting StartedPython setupRequires an installation of Python 3.8+.We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.The decorator is designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to thePrefect documentation.InstallationInstallprefect-alertwithpip:pipinstallprefect-alertCreate a notification blockBlocksenable you to store configuration and provide an interface for interacting with external systems.First, start with creating anotification blockthroughUIor a Python script:fromprefect.blocks.notificationsimportSlackWebhookslack_block=SlackWebhook(url="https://hooks.slack.com/services/XXX/XXX/XXX")Send an alertNext, use the block created and the decoratorprefect_alert.alert_on_failureto send alert when a flow fails.Send an alert when a flow failsfromprefectimportflow,taskfromprefect.blocks.notificationsimportSlackWebhookfromprefect_alertimportalert_on_failure@taskdefmay_fail():raiseValueError()@alert_on_failure(block_type=SlackWebhook,block_name="test")@flowdeffailed_flow():res=may_fail()returnresif__name__=="__main__":failed_flow()And you will see something like this on your Slack:Send an alert when a asynchronous flow failsfromprefectimportflow,taskfromprefect.blocks.notificationsimportSlackWebhookfromprefect_alertimportalert_on_failureimportasyncio@taskasyncdefmay_fail():raiseValueError()@alert_on_failure(block_type=SlackWebhook,block_name="test")@flowasyncdeffailed_flow():res=awaitmay_fail()returnresif__name__=="__main__":asyncio.run(failed_flow())ResourcesIf you encounter any bugs while usingprefect-alert, feel free to open an issue in theprefect-alertrepository.If you have any questions or issues while usingprefect-alert, you can find help in either thePrefect Discourse forumor thePrefect Slack community.Feel free to ⭐️ or watchprefect-alertfor updates too!DevelopmentIf you'd like to install a version ofprefect-alertfor development, clone the repository and perform an editable install withpip:gitclonehttps://github.com/khuyentran1401/prefect-alert.gitcdprefect-alert/

pipinstall-e".[dev]"# Install linting pre-commit hookspre-commitinstall
Package: prefect-aws
prefect-awsWelcome!prefect-awsmakes it easy to leverage the capabilities of AWS in your flows, featuring support for ECS, S3, Secrets Manager, and Batch.Visit the full docshere.InstallationTo start usingprefect-aws:pipinstallprefect-awsContributingThanks for thinking about chipping in! Check out thisstep-by-step guideon how to get started.
Package: prefect-aws-batch
Prefect AWS BatchUtilities for interacting with AWS BatchFree software: Apache Software License 2.0Documentation:https://prefect-aws-batch.readthedocs.io.FeaturesTODOCreditsThis package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template.History0.1.0 (2022-10-03)First release on PyPI.
Package: prefect-client
PrefectPrefect is an orchestration and observability platform for building, observing, and triaging workflows.
It's the simplest way to transform Python code into an interactive workflow application.Prefect allows you to expose your workflows through an API so teams dependent on you can programmatically access your pipelines, business logic, and more.
Prefect also allows you to standardize workflow development and deployment across your organization.With Prefect, you can build resilient, dynamic workflows that react to the world around them and recover from unexpected changes.
With just a few decorators, Prefect supercharges your code with features like automatic retries, distributed execution, scheduling, caching, and much more.Every activity is tracked and can be monitored with a self-hostedPrefect serverinstance or managedPrefect Clouddashboard.Getting startedPrefect requires Python 3.8 or later. Toinstall Prefect, run the following command:pipinstallprefectThen create and run a Python file that uses Prefectflowandtaskdecorators to orchestrate and observe your workflow that fetches the number of GitHub stars from a repository.fromprefectimportflow,taskfromtypingimportListimporthttpx@task(log_prints=True)defget_stars(repo:str):url=f"https://api.github.com/repos/{repo}"count=httpx.get(url).json()["stargazers_count"]print(f"{repo}has{count}stars!")@flow(name="GitHub Stars")defgithub_stars(repos:List[str]):forrepoinrepos:get_stars(repo)# run the flow!if__name__=="__main__":github_stars(["PrefectHQ/Prefect"])Fire up the Prefect UI to see what happened:prefectserverstartTo run your workflow on a schedule, turn it into a deployment and schedule it to run every minute by changing the last line of your script to the following:github_stars.serve(name="first-deployment",cron="* * * * *")You now have a server running locally that is looking for scheduled deployments!
Additionally you can run your workflow manually from the UI or CLI - and if you're using Prefect Cloud, you can even run deployments in response toevents.Prefect CloudStop worrying about your workflows.
Prefect Cloud allows you to centrally deploy, monitor, and manage the data workflows you support. With managed orchestration, automations, and webhooks, all backed by enterprise-class security, build production-ready code quickly and reliably.Read more about Prefect Cloudhereor sign up totry it for yourself.Next stepsThere's lots more you can do to orchestrate and observe your workflows with Prefect!
Start with ourfriendly tutorialor explore thecore concepts of Prefect workflows.Join the communityPrefect is made possible by the fastest growing community of thousands of friendly data engineers. Join us in building a new kind of workflow system. ThePrefect Slack communityis a fantastic place to learn more about Prefect, ask questions, or get help with workflow design. All community forums, including code contributions, issue discussions, and slack messages are subject to ourCode of Conduct.ContributeSee ourdocumentation on contributing to Prefect.Thanks for being part of the mission to build a new kind of workflow system and, of course,happy engineering!
Package: prefect-cloud-manager
prefect-managerOverviewThis is a CLI tool that allows to switch between multiple Prefect accounts
with little effort. It is a personal project of mine that I use to switch between a work Prefect account and a
Prefect account I use for my own personal projects.This is not a tool meant to support large teams with many agents running (although it definitely could.)All the files used by this tool can be found in~/.prefect-manager.If you haven't check outPrefectit is an amazing tool for orchestrating data and ML pipelines.
I would highly recommend checking it out!Installationpip install prefect-cloud-managerUsageThe CLI has a few basic commandsHelpTo see the command options simply type:prefect-managerAdd AccountTo add a new account and its access key:prefect-manager add-accountThen you will be prompted to enter the correct informationList AccountsTo see all of the available accounts:prefect-manager list-accountsActivate AccountTo activate an account you have added:prefect-manager activate {ACCOUNT_NAME}Add New ConfigurationThere are a lot of possible options in the config.toml file that prefect
uses. You are able to track multiple configurations if you want to use different
settings with different accounts. For more information on this see thePrefect documentationTo add an existing config to the tracked configs:prefect-manager add-configs {CONFIG_NAME} {PATH_TO_CONFIG}List Tracked Config FilesTo see all the configs you have saved:prefect-manager list-configsEdit ConfigTo edit a config you already have using nano:prefect-manager edit-config
Package: prefect-collection
prefect-collectionWelcome to your new Prefect collection! Feel free to modify this README to fit the needs of your collection!Getting StartedPython setupRequires an installation of Python 3.7+We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.Verify collection bootstrapTo ensure that your collection bootstrap was successful you can perform the following steps:Create an editable install of your projectpipinstall-e".[dev]Run the tests for the example tasks and flow in the bootstrapped collectionpytesttestsServe the bootstrapped example docs withmkdocsmkdocsserveYou are now ready to start creating your Prefect collection!Developing tasks and flowsFor information about the use and development of tasks and flow, check out theflowsandtasksconcepts docs in the Prefect docs.Writing documentationThis collection has been setup to withmkdocsfor automatically generated documentation. The signatures and docstrings of your tasks and flow will be used to generate documentation for the users of this collection. You can make changes to the structure of the generated documentation by editing themkdocs.ymlfile in this project.Development lifecycleCI PipelineThis collection comes withGitHub Actionsfor testing and linting. To add additional actions, you can add jobs in the.github/workflowsfolder. On PR, the pipeline will run linting viablackandflake8and unit tests viapytest.Package and PublishGitHub actions will also handle packaging and publishing of your collection toPyPIso other Prefect users can your collection in their flows.In order to publish to PyPI, you'll need a PyPI account and generate an API token to authenticate with PyPI when publishing new versions of your collection. ThePyPI documentationoutlines the steps needed to get an API token.Once you've obtained a PyPI API token,create a GitHub secretnamedPYPI_API_TOKEN.There is also a GitHub Action that performs a test publish totest PyPIto allow for a test deployment without affecting PyPI. You will need a separate token for test PyPI which can be saved in a GitHub secret namesTEST_PYPI_API_TOKENFurther guidanceIf you run into any issues during the bootstrapping process, feel free to open an issue in theprefect-collection-templaterepository.If you have any questions or issues while developing your collection, you can find help in either thePrefect Discourse forumor thePrefect Slack community
Package: prefect-cubejs
prefect-cubejsWelcome!Prefect collection of tasks to interact withCube.jsGetting StartedPython setupRequires an installation of Python 3.7+.We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.These tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to thePrefect documentation.InstallationInstallprefect-cubejswithpip:pipinstallprefect-cubejsWrite and run a flowfromprefectimportflowfromprefect_cubejs.tasksimport(run_query)@flowdefexample_flow():run_query(subdomain="<your Cube Cloud subdomain>",api_secret="<your Cube.js API secret>",query="<a Cube.js JSON formatted query>")example_flow()ResourcesIf you encounter any bugs while usingprefect-cubejs, feel free to open an issue in theprefect-cubejsrepository.If you have any questions or issues while usingprefect-cubejs, you can find help in either thePrefect Discourse forumor thePrefect Slack community.DevelopmentIf you'd like to install a version ofprefect-cubejsfor development, clone the repository and perform an editable install withpip:gitclonehttps://github.com/AlessandroLollo/prefect-cubejs.gitcdprefect-cubejs/

pipinstall-e".[dev]"# Install linting pre-commit hookspre-commitinstall
Package: prefect-dbtcloud
prefect-dbtcloudWelcome!Collection of tasks to interact with dbt CloudGetting StartedPython setupRequires an installation of Python 3.7+.We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.These tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to thePrefect documentation.InstallationInstallprefect-dbtcloudwithpip:pipinstallprefect-dbtcloudWrite and run a flowfromprefectimportflowfromprefect_dbtcloud.tasksimport(run_job)@flowdeftrigger_dbt_cloud_job_run():run_job(cause="<the motivation to run the job>",account_id=<yourdbtCloudAccountID>,job_id=<theidentifierofthejobtorun>,token="<your dbt Cloud API token>",wait_for_job_run_completion=False)trigger_dbt_cloud_job_run()ResourcesIf you encounter any bugs while usingprefect-dbtcloud, feel free to open an issue in theprefect-dbtcloudrepository.If you have any questions or issues while usingprefect-dbtcloud, you can find help in either thePrefect Discourse forumor thePrefect Slack community.DevelopmentIf you'd like to install a version ofprefect-dbtcloudfor development, clone the repository and perform an editable install withpip:gitclonehttps://github.com/AlessandroLollo/prefect-dbtcloud.gitcdprefect-dbtcloud/

pipinstall-e".[dev]"# Install linting pre-commit hookspre-commitinstall
Package: prefect-dbt-flow
prefect-dbt-flowprefect-dbt-flow is a Python library that enables Prefect to convert dbt workflows into independent tasks within a Prefect flow. This integration simplifies the orchestration and execution of dbt models and tests using Prefect, allowing you to build robust data pipelines and monitor your dbt projects efficiently.dbt is an immensely popular tool for building and testing data transformation models, and Prefect is a versatile workflow management system. This integration brings together the best of both worlds, empowering data engineers and analysts to create robust data pipelines.Key features:Simplified Orchestration: Define and manage your dbt projects and models as Prefect tasks, creating a seamless pipeline for data transformation.Monitoring and Error Handling: Gain deep insights into the execution of your dbt workflows and take immediate action in case of issues.Workflow Consistency: Ensure your dbt workflows run consistently by managing them through Prefect. This consistency is crucial for maintaining data quality and reliability.Advanced Configuration: Customize your dbt workflow by adjusting the dbt project, profile, and DAG options. You can also use Prefect features like scheduling, notifications, and task retries to monitor and manage your dbt flows effectively.To get started, check out ourgetting started guide.Active Development Notice:prefect-dbt-flow is actively under development and may not be ready for production use. We advise users to be aware of potential breaking changes as the library evolves. Please check the changelog for updates.How to InstallYou can install prefect-dbt-flow via pip:pipinstallprefect-dbt-flowNote: prefect-dbt-flow does not come with dbt as a dependency. You will need to install dbt or a dbt-adapter separately.Basic UsageHere's an example of how to use prefect-dbt-flow to create a Prefect flow for your dbt project:fromprefect_dbt_flowimportdbt_flowfromprefect_dbt_flow.dbtimportDbtProfile,DbtProjectmy_flow=dbt_flow(project=DbtProject(name="jaffle_shop",project_dir="path_to/jaffle_shop",profiles_dir="path_to/jaffle_shop",),profile=DbtProfile(target="dev",overrides={"type":"duckdb","path":"path_to/duckdb.db",},),)if__name__=="__main__":my_flow()For more information consult thedocsInspirationprefect-dbt-flow draws inspiration from various projects in the data engineering and workflow orchestration space, including:astronomer-cosmosdbt + Dagsterprefect-dataplatform - Anna GellerLicenseThis project is licensed under the MIT License. You are free to use, modify, and distribute this software as per the terms of the license. If you find this project helpful, please consider giving it a star on GitHub.
Package: prefect-discord
Prefect DiscordPrefect Discord is a lightweight package to send notifications from flows and tasks to a Discord channel.
The alerting system is configurable and allows many options that suit different flows and tasks.Usage Examplefromprefectimporttaskfromprefect_discordimportdiscord_notifier@task(state_handlers=[discord_notifier(ignore_states=[Running])])defadd(x,y):returnx+yConfigurationEnvironment Variablesprefect_discordexpects a few secrets set on Prefect'sconfig.toml:DISCORD_WEBHOOK_URL= "discord_channel_webhook_url"DISCORD_WEBHOOK_THUMBNAIL_URL= "url_for_the_main_image"DISCORD_WEBHOOK_FOOTER_MESSAGE= "footer_text_next_to_icon"DISCORD_WEBHOOK_FOOTER_ICON_URL= "url_for_the_footer_image"State Handler ParametersThe state handler can also be configured with some optionsignore_states ([State], optional): list ofStateclasses to ignore, e.g.,[Running, Scheduled]. Ifnew_stateis an instance of one of the passed states,
no notification will occur.only_states ([State], optional): similar toignore_states, but insteadonlynotifies you if the Task / Flow is in a state from the provided list ofStateclasseswebhook_secret (str, optional): the name of the Prefect Secret that stores your Discord
webhook URL; defaults to"DISCORD_WEBHOOK_URL"backend_info (bool, optional): Whether to supply the Discord notification with urls
pointing to backend pages; defaults to Trueproxies (dict), optional):dictwith "http" and/or "https" keys, passed torequests.post- for situations where a proxy is required to send requests to the
Discrd webhook