Saravanan666/pypi-org huggingface dataset:
Package: prefect-lakefs
prefect-lakefsVisit the full docshereto see additional examples and the API reference.Prefect integrations for interacting with LakeFS services.Welcome!prefect-lakefsis a collection of Prefect tasks and flows which can be used to interact with lakeFS managed datalakes.Jump toexamples.ResourcesFor more tips on how to use tasks and flows in a Collection, check outUsing Collections!InstallationInstallprefect-lakefswithpip:pipinstallprefect-lakefsRequires an installation of Python 3.8+.We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.These tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to thePrefect documentation.Example UsageConfigure LakeFS Credentials and dive in.importasynciofromprefect_lakefs.credentialsimportLakeFSCredentialsfromprefect_lakefs.tasksimportlist_branches# You can configure this while adding a block in the prefect-ui or#   you can save the block using .save() utility method provided by the block.lakefs_creds=LakeFSCredentials(endpoint_url="http://localhost:8000/api/v1",access_key_id="AKIAIOSFODNN7EXAMPLE",secret_access_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",).save("lakefs-creds")@flowdeflist_branches_for_example_repo():branches=list_branches(lakefs_credentials=LakeFSCredentials.load("lakefs-creds"),repository="example",)print(branches)if__name__=="__main__":# run the flowasyncio.run(list_branches())FeedbackIf you encounter any bugs while usingprefect-lakefs, feel free to open an issue in theprefect-lakefsrepository.If you have any questions or issues while usingprefect-lakefs, you can find help in either thePrefect Discourse forumor thePrefect Slack communityor theLakeFS Slack Community.Feel free to star or watchprefect-lakefsfor updates too!ContributingIf you'd like to help contribute to fix an issue or add a feature toprefect-lakefs, pleasepropose changes through a pull request from a fork of the repository.Here are the steps:Fork the repositoryClone the forked repositoryInstall the repository and its dependencies:pip install -e ".[dev]"Make desired changesAdd testsInsert an entry toCHANGELOG.mdInstallpre-committo perform quality checks prior to commit:pre-commit installgit commit,git push, and create a pull request
Package: prefect-memory-profiling
prefect-memory-profilingWelcome!A collection of Prefect Tasks for memory profiling in Prefect 1.0.Getting StartedPython setupRequires an installation of Python 3.7+.We recommend using a Python virtual environment manager such aspipenv,condaorvirtualenv.For more information about how to use Prefect, please refer to thePrefect documentation.InstallationInstallprefect-memory-profilingwithpip:pipinstallprefect-memory-profilingWrite and run a flowfromprefectimportFlowfromprefect_memory_profiling.tasksimportprofiled_task@profiled_task(name='My Profiled Task',stream=open('logfile.txt','a+'))defresource_intensive_task(n:int=100):a=[n**nforninrange(2*n)]b=[n**nforninrange(4*n)]c=[n**nforninrange(n**2)]returnsum(a+b+c)withFlow('Memory Profiled Flow')asflow:resource_intensive_task()if__name__=="__main__":flow.run(run_on_schedule=False)ResourcesIf you encounter any bugs while usingprefect-memory-profiling, feel free to open an issue in theprefect-memory-profilingrepository.If you have any questions or issues while usingprefect-memory-profiling, you can find help in either thePrefect Discourse forumor thePrefect Slack community.DevelopmentIf you'd like to install a version ofprefect-memory-profilingfor development, clone the repository and perform an editable install withpip:gitclonehttps://github.com/zzstoatzz/prefect-memory-profiling.gitcdprefect-memory-profiling/

pipinstall-e".[dev]"# Install linting pre-commit hookspre-commitinstall
Package: prefect-metricflow
prefect-metricflowWelcome!Collection of tasks to interact with MetricFlowGetting StartedPython setupRequires an installation of Python 3.8 or 3.9.We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.These tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to thePrefect documentation.InstallationInstallprefect-metricflowwithpip:pipinstallprefect-metricflowWrite and run a flowfromprefectimportflowfromprefect_metricflow.tasksimport(materialize)@flowdefcreate_materialization_with_metricflow():returnmaterialize(materialization_name="my_materialization",config={"dwh_dialect":"redshift","dwh_host":"host","dwh_port":5439,"dwh_user":"dw_user","dwh_password":"dw_pwd","dwh_database":"dw_db","dwh_schema":"dw_schema","model_path":"path/to/models",})create_materialization_with_metricflow()@flowdefdrop_materialization_with_metricflow():returndrop_materialization(materialization_name="my_materialization",config={"dwh_dialect":"redshift","dwh_host":"host","dwh_port":5439,"dwh_user":"dw_user","dwh_password":"dw_pwd","dwh_database":"dw_db","dwh_schema":"dw_schema","model_path":"path/to/models",})drop_materialization_with_metricflow()ResourcesIf you encounter any bugs while usingprefect-metricflow, feel free to open an issue in theprefect-metricflowrepository.If you have any questions or issues while usingprefect-metricflow, you can find help in either thePrefect Discourse forumor thePrefect Slack community.DevelopmentIf you'd like to install a version ofprefect-metricflowfor development, clone the repository and perform an editable install withpip:gitclonehttps://github.com/AlessandroLollo/prefect-metricflow.gitcdprefect-metricflow/

pipinstall-e".[dev]"# Install linting pre-commit hookspre-commitinstall
Package: prefect-monday
prefect-mondayWelcome!Prefect integrations interacting with monday.com.The tasks within this collection were created by a code generator using the service's GraphQL schema.Getting StartedPython setupRequires an installation of Python 3.7+.We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.These tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to thePrefect documentation.InstallationInstallprefect-mondaywithpip:pipinstallprefect-mondayThen, register toview the blockon Prefect Cloud:prefectblockregister-mprefect_monday.credentialsQuery personal account infoNote, to use theloadmethod, you must already have a block documentsaved through codeorsaved through the UI.fromprefectimportflowfromprefect_monday.credentialsimportMondayCredentialsfromprefect_monday.meimportquery_me@flowdefquery_me_flow():monday_credentials=MondayCredentials.load("BLOCK_NAME")result=query_me(monday_credentials)returnresultquery_me_flow()Query available boardsfromprefectimportflowfromprefect_monday.credentialsimportMondayCredentialsfromprefect_monday.boardsimportquery_boards@flowdefquery_boards_flow():monday_credentials=MondayCredentials.load("BLOCK_NAME")boards=query_boards(monday_credentials=monday_credentials)returnboardsquery_boards_flow()Create new workspacefromprefectimportflowfromprefect_monday.credentialsimportMondayCredentialsfromprefect_monday.mutationsimportcreate_workspacefromprefect_monday.schemasimportgraphql_schema@flowdefcreate_workspace_flow():monday_credentials=MondayCredentials.load("BLOCK_NAME")workspace=create_workspace("integrations-test-workspace",graphql_schema.WorkspaceKind.open,monday_credentials=monday_credentials)returnworkspacecreate_workspace_flow()ResourcesIf you encounter any bugs while usingprefect-monday, feel free to open an issue in theprefect-mondayrepository.If you have any questions or issues while usingprefect-monday, you can find help in either thePrefect Discourse forumor thePrefect Slack community.Feel free to ⭐️ or watchprefect-mondayfor updates too!DevelopmentIf you'd like to install a version ofprefect-mondayfor development, clone the repository and perform an editable install withpip:gitclonehttps://github.com/PrefectHQ/prefect-monday.gitcdprefect-monday/

pipinstall-e".[dev]"# Install linting pre-commit hookspre-commitinstall
Package: prefect-multiprocess
Prefect Multiprocess Task RunnerWelcome!This Prefect collection contains a multiprocess task runner. It is ideal for running CPU-intensive Prefect tasks in parallel. It is useful in scenarios where you want to spread computation across multiple CPU cores on a single machine without adding heavy dependencies like Dask. This package does not require any extra dependencies beyond what Prefect already installs.Getting StartedInstall the package by running:pip install prefect-multiprocessThen, use the task runner in your Prefect flows. The task runner only accepts one parameter:processes, which controls the number of worker processes to start for running tasks. If not provided, it defaults to the number of CPUs on the host machine.Examples:Create one process for every CPU/CPU corefromprefectimportflowfromprefect_multiprocess.task_runnersimportMultiprocessTaskRunner@flow(task_runner=MultiprocessTaskRunner())defmy_flow():...Customizing the number of processes:@flow(task_runner=MultiprocessTaskRunner(processes=4))defmy_flow():...Python setupRequires an installation of Python 3.8+.We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.These tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to thePrefect documentation.LimitationsMultiprocessTaskRunnerusescloudpickleto serialize tasks and return values, so task parameters and returns need to be values thatcloudpicklecan handle.ResourcesIf you encounter any bugs while usingprefect-multiprocess, feel free to open an issue in theprefect-multiprocessrepository.Feel free to ⭐️ or watchprefect-multiprocessfor updates too!DevelopmentIf you'd like to install a version ofprefect-multiprocessfor development, clone the repository and perform an editable install withpip:gitclonehttps://github.com/rpeden/prefect-multiprocess.gitcdprefect-multiprocess/

pipinstall-e".[dev]"# Install linting pre-commit hookspre-commitinstall
Package: prefect-openmetadata
prefect-openmetadataWelcome!UsingPrefectandOpenMetadatatogether will help you build and maintain adata platform you can trust.Prefect allows you to coordinate your dataflow and provides visibility into the health of yourworkflow executionandworkflow lineage. With OpenMetadata integration, you can enrich your orchestration system with metadata about data lineage, data catalog, data quality and governance, giving you more information about the health of your system.Getting StartedPython setupRequires an installation of Python 3.8+.We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.InstallationInstallprefect-openmetadatawithpip:pipinstallprefect-openmetadataInstallOpenMetadataandPrefectHead over to theinstall OpenMetadatapage for detailed instructions on how to install and configure both platforms.Write and run metadata ingestion flowfromprefect_openmetadata.flowsimportingest_metadataconfig="""See an example in the section: Run ingestion flow"""if__name__=="__main__":ingest_metadata(config)For more details, check therun ingestion flowsection.Schedule a metadata ingestion flowSimple example:prefectdeploymentbuild-a-ndevmyflow.py:ingest_metadata--interval900For more details, check theschedule ingestion flowsection.ResourcesIf you encounter any bugs while usingprefect-openmetadata, feel free to open an issue in theprefect-openmetadatarepository.If you have any questions or issues while usingprefect-openmetadata, you can find help in either thePrefect Discourse forumor thePrefect Slack community.DevelopmentIf you'd like to install a version ofprefect-openmetadatafor development, clone the repository and perform an editable install withpip:gitclonehttps://github.com/PrefectHQ/prefect-openmetadata.gitcdprefect-openmetadata/

pipinstall-e".[dev]"# Install linting pre-commit hookspre-commitinstall
Package: prefect-opentelemetry
prefect-opentelemetryWelcome!Extend prefect Orion withOpentelemetrycapability to monitor the server sideGetting StartedThis project create a new Perfect orion factory that wrap the original one and add opentelemetry capability for the following areaFastAPISQLAlchemyPostgresSQLLiteThis is setup to only export all the opentelemetry at the format OTLP to an exporter that you can use after to dispatch to your metrics, log or trace.Run SampleThe project has a docker compose file that allow you to start a server with PostgreSQL which will push the trace to Zipkin. The sample is using OpenTelemetry-Collector as a buffer. This allow you to push your trace to any standard trace likezipkin,jeagerortempoSample view from zipkinPython setupRequires an installation of Python 3.8+ .We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.These tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to thePrefect documentation.NoteBecause this change the base factory that create FastAPI app, you cannot useprefect orion startStart Prefect server using manual command of uvicornuvicorn --app-dir $(python -c 'import prefect; print(prefect.__module_path__.parent)')  --factory  prefect_opentelemetry.server:create_app_with_OTLP --host 0.0.0.0 --port 4200The part$(python -c 'import prefect; print(prefect.__module_path__.parent)')allow to setup the app-dir to ensure the UI can work.The setup of the server is done using env var. You can find the list from prefecthere. The import one to setup arePREFECT_ORION_DATABASE_CONNECTION_URL: Coonection string to the databaseOTEL_EXPORTER_OTLP_TRACES_ENDPOINT: URL to the OTLP colletor using gRPCPREFECT_ORION_SERVICES_SCHEDULER_ENABLED: To only enable the schedulerPREFECT_ORION_ANALYTICS_ENABLED: To enable AnalyticPREFECT_ORION_SERVICES_LATE_RUNS_ENABLED: To enable late runPREFECT_ORION_SERVICES_UI: To enable the UIResourcesIf you encounter any bugs while usingprefect-opentelemetry, feel free to open an issue in theprefect-opentelemetryrepository.If you have any questions or issues while usingprefect-opentelemetry, you can find help in either thePrefect Discourse forumor thePrefect Slack community.DevelopmentIf you'd like to install a version ofprefect-opentelemetryfor development, clone the repository and perform an editable install withpip:gitclonehttps://github.com/davzucky/prefect-opentelemetry.gitcdprefect-opentelemetry/

pipinstall-e".[dev]"# Install linting pre-commit hookspre-commitinstall
Package: prefect-pokemon
prefect-pokemonWelcome!Tasks for retrieving Pokemon informationGetting StartedPython setupRequires an installation of Python 3.7+.We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.These tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to thePrefect documentation.InstallationInstallprefect-pokemonwithpip:pipinstallprefect-pokemonWrite and run a flowfromprefectimportflowfromprefect_pokemon.tasksimport(goodbye_prefect_pokemon,hello_prefect_pokemon,)@flowdefexample_flow():hello_prefect_pokemongoodbye_prefect_pokemonexample_flow()ResourcesIf you encounter any bugs while usingprefect-pokemon, feel free to open an issue in theprefect-pokemonrepository.If you have any questions or issues while usingprefect-pokemon, you can find help in either thePrefect Discourse forumor thePrefect Slack community.DevelopmentIf you'd like to install a version ofprefect-pokemonfor development, clone the repository and perform an editable install withpip:gitclonehttps://github.com/desertaxle/prefect-pokemon.gitcdprefect-pokemon/

pipinstall-e".[dev]"# Install linting pre-commit hookspre-commitinstall
Package: prefect-redis
prefect-redisWelcome!Prefect integrations for working with RedisGetting StartedPython setupRequires an installation of Python 3.7+.We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.These tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to thePrefect documentation.InstallationInstallprefect-rediswithpip:pipinstallprefect-redisThen, register toview the blockon Prefect Cloud:prefectblockregister-mprefect_redis.credentialsNote, to use theloadmethod on Blocks, you must already have a block documentsaved through codeorsaved through the UI.Write and run a flowfromprefectimportflowfromprefect_redisimport(RedisCredentials,redis_set,redis_get,)@flowdefexample_flow():# Load credentials-blockcredentials=RedisCredentials.load("my-redis-store")# Set a redis-key - Supports any object that is not a live connectionredis_set(credentials,"mykey",{"foo":"bar"})# Get a redis keyval=redis_get(credentials,"mykey")print(val)example_flow()ResourcesIf you encounter any bugs while usingprefect-redis, feel free to open an issue in theprefect-redisrepository.If you have any questions or issues while usingprefect-redis, you can find help in either thePrefect Discourse forumor thePrefect Slack community.Feel free to ⭐️ or watchprefect-redisfor updates too!DevelopmentIf you'd like to install a version ofprefect-redisfor development, clone the repository and perform an editable install withpip:gitclonehttps://github.com/C4IROcean/prefect-redis.gitcdprefect-redis/

pipinstall-e".[dev]"# Install linting pre-commit hookspre-commitinstall
Package: prefect-saturn
prefect-saturnprefect-saturnis a Python package that makes it easy to runPrefect Cloudflows on a Dask cluster withSaturn Cloud. For a detailed tutorial, see"Fault-Tolerant Data Pipelines with Prefect Cloud
".Installationprefect-saturnis available on PyPi.pipinstallprefect-saturnprefect-saturncan be installed directly from GitHubpipinstallgit+https://github.com/saturncloud/prefect-saturn.git@mainGetting Startedprefect-saturnis intended for use inside aSaturn Cloudenvironment, such as a Jupyter notebook.importprefectfromprefectimportFlow,taskfromprefect_saturnimportPrefectCloudIntegration@taskdefhello_task():logger=prefect.context.get("logger")logger.info("hello prefect-saturn")flow=Flow("sample-flow",tasks=[hello_task])project_name="sample-project"integration=PrefectCloudIntegration(prefect_cloud_project_name=project_name)flow=integration.register_flow_with_saturn(flow)flow.register(project_name=project_name,labels=["saturn-cloud"])Customize DaskYou can customize the size and behavior of the Dask cluster used to run prefect flows.prefect_saturn.PrefectCloudIntegration.register_flow_with_saturn()accepts to arguments to accomplish this:dask_cluster_kwargs: keyword arguments to pass to the constructordask_saturn.SaturnCluster.dask_adapt_kwargs: keyword arguments used to configure"Adaptive Scaling"For example, the code below tells Saturn that this flow should run on a Dask cluster with 3 xlarge workers, and that prefect should shut down the cluster once the flow run has finished.flow=integration.register_flow_with_saturn(flow=flow,dask_cluster_kwargs={"n_workers":3,"worker_size":"xlarge","autoclose":True})flow.register(project_name=project_name,labels=["saturn-cloud"])ContributingSeeCONTRIBUTING.mdfor documentation on how to test and contribute toprefect-saturn.
Package: prefect-sifflet
prefect-siffletWelcome!Integrate Sifflet Data Observability Platform with PrefectGetting StartedPython setupRequires an installation of Python 3.7+.We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.These tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to thePrefect documentation.InstallationInstallprefect-siffletwithpip:pipinstallprefect-siffletThen, register toview the blockon Prefect Cloud:prefectblockregister-mprefect_sifflet.credentialsNote, to use theloadmethod on Blocks, you must already have a block documentsaved through codeorsaved through the UI.Write and run a flowfromprefectimportflowfromprefect_sifflet.tasksimport(trigger_sifflet_rule_run,get_sifflet_rule_run,)@flowdefexecute_rule():tenant="<your tenant>"api_token="<your API token>"rule_id="<your rule ID>"response=trigger_sifflet_rule_run(tenant=tenant,api_token=api_token,rule_id=rule_id,wait_for_completion=False)rule_run_id=response["id"]rule_run_result=get_sifflet_rule_run(tenant=tenant,api_token=api_token,rule_id=rule_id,rule_run_id=rule_run_id)execute_rule()ResourcesIf you encounter any bugs while usingprefect-sifflet, feel free to open an issue in theprefect-siffletrepository.If you have any questions or issues while usingprefect-sifflet, you can find help in either thePrefect Discourse forumor thePrefect Slack community.Feel free to ⭐️ or watchprefect-siffletfor updates too!DevelopmentIf you'd like to install a version ofprefect-siffletfor development, clone the repository and perform an editable install withpip:gitclonehttps://github.com/Siffletapp/prefect-sifflet.gitcdprefect-sifflet/

pipinstall-e".[dev]"# Install linting pre-commit hookspre-commitinstall
Package: prefect-sns
prefect-snsVisit the full docshereto see additional examples and the API reference.Prefect block for integration with AWS SNSResourcesFor more tips on how to use tasks and flows in a Collection, check outUsing Collections!InstallationInstallprefect-snswithpip:pipinstallprefect-snsRequires an installation of Python 3.7+.We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.These tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to thePrefect documentation.FeedbackIf you encounter any bugs while usingprefect-sns, feel free to open an issue in theprefect-snsrepository.If you have any questions or issues while usingprefect-sns, you can find help in either thePrefect Discourse forumor thePrefect Slack community.Feel free to star or watchprefect-snsfor updates too!ContributingIf you'd like to help contribute to fix an issue or add a feature toprefect-sns, pleasepropose changes through a pull request from a fork of the repository.Here are the steps:Fork the repositoryClone the forked repositoryInstall the repository and its dependencies:pip install -e ".[dev]"Make desired changesAdd testsInsert an entry toCHANGELOG.mdInstallpre-committo perform quality checks prior to commit:pre-commit installgit commit,git push, and create a pull request