Saravanan666/pypi-org huggingface dataset:
Package: airflow-provider-hightouch
Apache Airflow Provider for HightouchProvides an Airflow Operator and Hook forHightouch.
This allows the user to initiate a run for a sync from Airflow.InstallationPre-requisites: An environment runningapache-airflow>= 1.10, including >= 2.pip install airflow-provider-hightouchConfigurationIn the Airflow Connections UI, create a new connection for Hightouch.Conn ID:hightouch_defaultConn Type:HTTPHost:https://api.hightouch.comPassword: enter the API key for your workspace. You can generate an API
key from yourWorkspace SettingsThe Operator uses thehightouch_defaultconnection id by default, but
if needed, you can create additional Airflow Connections and reference them
in the operatorModulesHightouchTriggerSyncOperatorStarts a Hightouch Sync Run. Requires thesync_idor thesync_slugfor the sync you wish to
run.Returns thesync_run_idof the sync it triggers.The run is synchronous by default, and the task will be marked complete once the
sync is successfully completed.However, you can request a asynchronous request instead by passingsynchronous=Falseto the operator.If the API key is not authorized or if the request is invalid the task will fail.
If a run is already in progress, a new run will be triggered following the
completion of the existing run.HightouchSyncRunSensorMonitors a Hightouch Sync Run. Requires thesync_idand thesync_run_idof the sync you wish to monitor.
To obtain thesync_run_idof a sync triggered in Airflow, we recommend using XComs to pass the return value
ofHightouchTriggerSyncOperator.ExamplesCreating a run is as simple as importing the operator and providing it with
a sync_id. Anexample dagis available as well.from airflow_provider_hightouch.operators.hightouch import HightouchTriggerSyncOperator

with DAG(....) as dag:
...
    my_task = HightouchTriggerSyncOperator(task_id="run_my_sync", sync_id="123")
    my_other_task = HightouchTriggerSyncOperator(task_id="run_my_sync", sync_slug="my-sync-slug")IssuesPlease submitissuesandpull requestsin our official repo:https://github.com/hightouchio/airflow-provider-hightouchWe are happy to hear from you, for any feedback please email the authors atpedram@hightouch.io.AcknowledgementsSpecial thanks toFivetranfor their provider andMarcos Marx's Airbyte
contribution in the core Airflow repo for doing this before we had to
so we could generously learn from their hard work.
Package: airflow-provider-kafka
Kafka Airflow ProviderAn airflow provider to:interact with kafka clustersread from topicswrite to topicswait for specific messages to arrive to a topicThis package currently contains3 hooks (airflow_provider_kafka.hooks) :admin_client.KafkaAdminClientHook- a hook to work against the actual kafka admin clientconsumer.KafkaConsumerHook- a hook that creates a consumer and provides it for interactionproducer.KafkaProducerHook- a hook that creates a producer and provides it for interaction4 operators (airflow_provider_kafka.operators) :await_message.AwaitKafkaMessageOperator- a deferable operator (sensor) that awaits to encounter a message in the log before triggering down stream tasks.consume_from_topic.ConsumeFromTopicOperator- an operator that reads from a topic and applies a function to each message fetched.produce_to_topic.ProduceToTopicOperator- an operator that uses a iterable to produce messages as key/value pairs to a kafka topic.event_triggers_function.EventTriggersFunctionOperator- an operator that listens for messages on the topic and then triggers a downstream function before going back to listening.1 triggerairflow_provider_kafka.triggers:await_message.AwaitMessageTriggerQuick startpip install airflow-provider-kafkaExample usages :basic read/write/sense on a topicevent listener patternFAQsWhy confluent kafka and not (other library) ?A few reasons: theconfluent-kafkalibrary is guaranteed to be 1:1 functional with librdkafka, is faster, and is maintained by a company with a commercial stake in ensuring the continued quality and upkeep of it as a product.Why not release this into airflow directly ?I could probably make the PR and get it through, but the airflow code base is getting huge and I don't want to burden the maintainers with code that they don't own for maintainence. Also there's been multiple attempts to get a Kafka provider in before and this is just faster.Why is most of the configuration handled in a dict ?Because that's howconfluent-kafkadoes it. I'd rather maintain interfaces that people already using kafka are comfortable with as a starting point - I'm happy to add more options/ interfaces in later but would prefer to be thoughtful about it to ensure that there difference between these operators and the actual client interface are minimal.Local DevelopmentUnit TestsUnit tests are located attests/unit, a kafka server isn't required to run these tests.
execute withpytestSetup on M1 MacInstalling on M1 chip means a brew install of thelibrdkafkalibrary before you canpip install confluent-kafkabrewinstalllibrdkafkaexportC_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/includeexportLIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/lib
pipinstallconfluent-kafka
Package: airflow-provider-lakefs
lakeFS airflow providerlakeFS airflow provider enables a smooth integration of lakeFS with airflow's DAGs.
"Use the lakeFS provider to create branches, commit objects, wait for files to be written, and more."For usage example, check out theexample DAGWhat is lakeFSlakeFS is an open source layer that delivers resilience and manageability to object-storage based data lakes.With lakeFS you can build repeatable, atomic and versioned data lake operations - from complex ETL jobs to data science and analytics.lakeFS supports AWS S3, Azure Blob Storage and Google Cloud Storage as its underlying storage service. It is API compatible with S3, and works seamlessly with all modern data frameworks such as Spark, Hive, AWS Athena, Presto, etc.For more information see theofficial lakeFS documentation.CapabilitiesDevelopment Environment for DataExperimentation- try tools, upgrade versions and evaluate code changes in isolation.Reproducibility- go back to any point of time to a consistent version of your data lake.Continuous Data IntegrationIngest new data safely by enforcing best practices- make sure new data sources adhere to your lake’s best practices such as format and schema enforcement, naming convention, etc.Metadata validation- prevent breaking changes from entering the production data environment.Continuous Data DeploymentInstantly revert changes to data- if low quality data is exposed to your consumers, you can revert instantly to a former, consistent and correct snapshot of your data lake.Enforce cross collection consistency- provide to consumers several collections of data that must be synchronized, in one atomic, revertible, action.Prevent data quality issues by enablingTesting of production data before exposing it to users / consumers.Testing of intermediate results in your DAG to avoid cascading quality issues.PublishingThe repository include GitHub workflow that is trigger on publish event and will build and push the package to PyPI.Use the following steps to release:Updatesetup.pywith the new package versionUpdateCHANGELOG.mdwith changes for the new releaseUse GitHub release, use semver vX.X.XCommunityStay up to date and get lakeFS support via:Slack(to get help from our team and other users).Twitter(follow for updates and news)YouTube(learn from video tutorials)Contact us(for anything)More informationlakeFS documentationIf you would like to contribute, check out ourcontributing guide.RoadmapLicensinglakeFS is completely free and open source and licensed under theApache 2.0 License.
Package: airflow-provider-mlflow
AnApache Airflowprovider to interact with MLflow using Operators and Hooks for the following:RegistryDeploymentsPyfunchttps://mlflow.org/docs/latest/index.htmlQuick StartInstall and update using pip:pipinstallairflow-provider-mlflowSetting up Connections:Connection Type: HTTPLocal MLflowHost:http://localhost(if running Airflwo in docker:http://host.docker.internal)Port: 5000Hosted with Username/PasswordConnection Type: HTTPHost: Your MLflow host URLLogin: Your MLflow usernamePassword: Your MLflow passwordDatabricksHost: Your Databricks host URL (https://<instance-name>.cloud.databricks.com)Login: ‘token’Password: Your Databricks tokenExamples can be found in theexample_dagsdirectory of the repo.ChangelogWe followSemantic Versioningfor releases.
CheckCHANGELOG.rstfor the latest changes.LicenseApache License 2.0
Package: airflow-provider-nessie
Nessie Airflow ProviderDocumentation:https://projectnessie.github.io/nessie_providerSource Code:https://github.com/projectnessie/nessie_providerUsageTo use in airflow install via pippip install airflow-provider-nessie. SeeNessie Documentationfor
instructions on starting and using a Nessie server.Operators and HooksTo interact with Nessie from an airflow DAG you have the following options:Nessie Hook: register as a connection w/ Airflow and store your Nessie url and credentialsCreate reference operator: Create a Branch or Tag as part of an airflow DAGDelete reference operator: Delete a Branch or Tag as part of an airflow DAGCommit operator: commit objects to the Nessie database on a given branchMerge operator: merge one branch into anotherThese can be seen in action by looking at theExample DAGs.
Thebasic_nessie.pyDAG shows each operator in action and thespark_nessie_iceberg.pyDAG shows a more complicated example of performing an iceberg
transaction in Nessie from the Spark operator.DevelopmentSetup environementYou should havePipenvinstalled. Then, you can install the dependencies with:pipenvinstall--devAfter that, activate the virtual environment:pipenvshellRun unit testsYou can run all the tests with:maketestAlternatively, you can runpytestyourself:pytestFormat the codeExecute the following command to applyisortandblackformatting:makeformatLicenseThis project is licensed under the terms of the Apache Software License 2.0.
Package: airflow-provider-openmldb
Airflow OpenMLDB ProviderOverviewAirflow OpenMLDB Provider supports connecting to OpenMLDB. Specifically, connect to the OpenMLDB API Server.Operators:OpenMLDBLoadDataOperatorOpenMLDBSelectIntoOperatorOpenMLDBDeployOperatorOpenMLDBSQLOperator: the underlying implementation of operators above. Support all sql.Only operators and a hook, no sensors.BuildTo build openmldb provider, follow the steps below:Clone the repo.cdinto provider directory.Runpython3 -m pip install build.Runpython3 -m buildto build the wheel.Find the .whl file in/dist/*.whl.How to useWrite the dag, using openmldb operators, refsimple openmldb operator dag example.Create the connection in airflow, the name isopenmldb_conn_idyou set.Trigger the dag.
Package: airflow-provider-paradime-dbt
airflow-provider-paradime-dbtThis is the provider for Paradime to run and manage dbt™ jobs in production. The provider enables interaction with Paradime’s Bolt scheduler and management APIs.UsageCreate a connectionGenerate your API key, secret and endpoint from Paradime Workspace settings.Create a connection in Airflow, as shown below.Create a DAGHere is one example:fromairflow.decoratorsimportdagfromparadime_dbt_provider.operators.paradimeimportParadimeBoltDbtScheduleRunArtifactOperator,ParadimeBoltDbtScheduleRunOperatorfromparadime_dbt_provider.sensors.paradimeimportParadimeBoltDbtScheduleRunSensorPARADIME_CONN_ID="your_paradime_conn_id"# Update this to your connection idBOLT_SCHEDULE_NAME="your_schedule_name"# Update this to your schedule name@dag(default_args={"conn_id":PARADIME_CONN_ID},)defrun_schedule_and_download_manifest():# Run the schedule and return the run id as the xcom return valuetask_run_schedule=ParadimeBoltDbtScheduleRunOperator(task_id="run_schedule",schedule_name=BOLT_SCHEDULE_NAME)# Get the run id from the xcom return valuerun_id="{{ task_instance.xcom_pull(task_ids='run_schedule') }}"# Wait for the schedule to complete before continuingtask_wait_for_schedule=ParadimeBoltDbtScheduleRunSensor(task_id="wait_for_schedule",run_id=run_id)# Download the manifest.json file from the schedule run and return the path as the xcom return valuetask_download_manifest=ParadimeBoltDbtScheduleRunArtifactOperator(task_id="download_manifest",run_id=run_id,artifact_path="target/manifest.json")# Get the path to the manifest.json file from the xcom return valueoutput_path="{{ task_instance.xcom_pull(task_ids='download_manifest') }}"task_run_schedule>>task_wait_for_schedule>>task_download_manifestrun_schedule_and_download_manifest()Refer to theexample DAGsin this repository for more examples.
Package: airflow-provider-pulumi
Pulumi Airflow ProviderAn airflow provider to:preview infrastructure resources before deploymentdeploy infrastructure resources via Pulumidestroy infrastructure resourcesThis package currently contains1 hook :airflow_provider_pulumi.hooks.automation.PulumiAutoHook- a hook to setup the Pulumi backend connection.4 operators :airflow_provider_pulumi.operators.base.BasePulumiOperator- the base operator for Pulumi.airflow_provider_pulumi.operators.preview.PulumiPreviewOperator- an operator that previews the deployment of infrastructure resources with Pulumi.airflow_provider_pulumi.operators.up.PulumiUpOperator- an operator that deploys infrastructure resources with Pulumi.airflow_provider_pulumi.operators.destroy.PulumiDestroyOperator- an operator that destroys infrastructure resources with Pulumi.RequirementsThese operators require the Pulumi client to be installed. Use the following script to install the Pulumi client in your Airflow environment:curl-fsSLhttps://get.pulumi.com|shexportPATH="$HOME/.pulumi/bin:$PATH"Quick startpip install airflow-provider-pulumi# example_pulumi_dag.pyfromdatetimeimportdatetimefromairflow.decoratorsimportdagfromairflow_provider_pulumi.operators.destroyimportPulumiDestroyOperatorfromairflow_provider_pulumi.operators.previewimportPulumiPreviewOperatorfromairflow_provider_pulumi.operators.upimportPulumiUpOperator@dag(schedule_interval=None,start_date=datetime(2022,1,1),tags=["example"],)defexample_pulumi():defcreate_s3_bucket():importpulumiimportpulumi_awsasaws# Creates an AWS resource (S3 Bucket)bucket=aws.s3.Bucket("my-bucket")# Exports the DNS name of the bucketpulumi.export("bucket_name",bucket.bucket_domain_name)preview_s3_create_task=PulumiPreviewOperator(task_id="preview_s3_create",pulumi_program=create_s3_bucket,stack_config={"aws:region":"us-west-2"},plugins={"aws":"v5.0.0"},)s3_create_task=PulumiUpOperator(task_id="s3_create",pulumi_program=create_s3_bucket,stack_config={"aws:region":"us-west-2"},plugins={"aws":"v5.0.0"},)s3_destroy_task=PulumiDestroyOperator(task_id="s3_destroy",pulumi_program=create_s3_bucket,stack_config={"aws:region":"us-west-2"},plugins={"aws":"v5.0.0"},)preview_s3_create_task>>s3_create_task>>s3_destroy_taskexample_pulumi_dag=example_pulumi()DevelopmentUnit TestsUnit tests are located attests, the Pulumi client is required to run these tests. Execute withpytest.
Package: airflow-provider-rabbitmq
RabbitMQ Provider for Apache AirflowConfigurationIn the Airflow user interface, configure a connection with theConn Typeset to RabbitMQ.
Configure the following fields:Conn Id: How you wish to reference this connection.
The default value israbbitmq_default.login: Login for the RabbitMQ server.password: Password for the RabbitMQ server.,port: Port for the RabbitMQ server, typically 5672.host: Host of the RabbitMQ server.vhost: The virtual host you wish to connect to.ModulesRabbitMQ OperatorTheRabbitMQOperatorpublishes a message to your specificed RabbitMQ server.Import into your DAG using:fromrabbitmq_provider.operators.rabbitmqimportRabbitMQOperatorRabbitMQ SensorTheRabbitMQSensorchecks a given queue for a message. Once it has found a message
the sensor triggers downstream proccesses in your DAG.Import into your DAG using:fromrabbitmq_provider.sensors.rabbitmqimportRabbitMQSensorTestingTo run unit tests, use:poetryrunpytest.A RabbitMQ instance is required to run the tests. Use the following command:dockerrun--rm-it--hostnamemy-rabbit-p15672:15672-p5672:5672rabbitmq:3-management
Package: airflow-providers-aliyun-rocketmq
Airflow Provider for Aliyun RocketMQExamplefromaliyun_rocketmq_provider.hooks.aliyun_rocketmqimportAliyunRocketMQHookmessage_push_topic=AliyunRocketMQHook(topic="message-push")message_push_topic.run("helloWorld",fail_silently=True)
Package: airflow-provider-sapiq
SAP IQ Provider for Apache AirflowTable of ContentsInstallationConfigurationLicenseInstallationpip install airflow-provider-sapiqConfigurationIn the Airflow user interface, configure a connection with theConn Typeset to SAP IQ.
Configure the following fields:Conn Id: How you wish to reference this connection.
The default value issapiq_default.userid: Login for the SAP IQ server.password: Password for the SAP IQ server.,port: Port for the SAP IQ server, typically 2638.host: Host of the SAP IQ server.ModulesSAP IQ OperatorTheSapIqOperatorexequtes SQL query on SAP IQ server.Import into your DAG using:fromsapiq_provider.operators.SapIqOperatorimportSapIqOperatorLicenseairflow-provider-sapiqis distributed under the terms of theApache-2.0license.
Package: airflow-providers-hive-zk
airflow-providers-hive-zkProvider for Hive for Airflow 2.X. It is using ZK discovery service to find Hive server.Build and install locally:python3 -m build

pip install airflow-providers-hive-zk --no-index --find-links file:///<path>/git/airflow_provider_hive_zk/dist/Start airflow:airflow webserverCheck if provider is registered and if new connection type has appeared.Install:pip install airflow-providers-hive-zk
Package: airflow-provider-sifflet
Sifflet Provider for Apache AirflowThis package provides operators and hook that integrateSiffletinto Apache Airflow.
All classes for this provider package are in thesifflet_providerPython package.InstallationYou can install this package on top of an existing Airflow 2.1+ installationpipinstallairflow-provider-siffletThe package supports the following python versions: 3.7, 3.8, 3.9, 3.10ConfigurationIn the Airflow user interface, you can configure a Connection for Sifflet inAdmin->Connections->Add a new record.You will need to fill out the following:Connection Id: sifflet_default
Connection Type: Sifflet
Sifflet Tenant: <your_tenant_name> (for SaaS deployement)
Sifflet Backend URL: <your_backend_url> (for Self-hosted deployment)
Sifflet Token: <your_sifflet_access_token><your_sifflet_access_token>: you can find more information on how to generate ithereOne of the following is required:<your_tenant_name>: if you access to Sifflet with "https://abcdef.siffletdata.com", then your tenant would beabcdef<your_backend_url>: full URL to the Sifflet backend on your deployment, for instance:https://sifflet-backend.mycompany.comModulesOperatorsSiffletDbtIngestOperatorSiffletDbtIngestOperatorsends your DBT artifacts to the Sifflet application.Example usage:fromsifflet_provider.operators.dbtimportSiffletDbtIngestOperatorsifflet_dbt_ingest=SiffletDbtIngestOperator(task_id="sifflet_dbt_ingest",input_folder="<path to dbt project folder>",target="prod",project_name="<dbt project name>",)SiffletRunRuleOperatorSiffletRunRuleOperatorRun one or several Sifflet rules - requires rule id(s).Example usage:fromsifflet_provider.operators.ruleimportSiffletRunRuleOperatorsifflet_run_rule=SiffletRunRuleOperator(task_id="sifflet_run_rule",rule_ids=["3e2e2687-cd20-11ec-b38b-06bb20181849","3e19eb3e-cd20-11ec-b38b-06bb20181849","3e1a86f1-cd20-11ec-b38b-06bb20181849","3e2e1fc3-cd20-11ec-b38b-06bb20181849",],error_on_rule_fail=True)
Package: airflow-provider-skypilot
Apache Airflow Provider for SkyPilotA provider you can utilize multiple clouds on Apache Airflow through SkyPilot.InstallationThe SkyPilot provider for Apache Airflow was developed and tested on an environment with the following dependencies installed:Apache Airflow>= 2.6.0SkyPilot>= 0.4.1The installation of the SkyPilot provider may start from the Airflow environment configured with Docker instructed in"Running Airflow in Docker".
Base on the docker configuration, add apip installcommand in the Dockerfile and build your own Docker image.RUNpipinstall--userairflow-provider-skypilotThen, make sure that SkyPilot is properly installed and initialized on the same environment. The initialization includes cloud account setup and access verification.
Please refer toSkyPilot Installationfor more information.ConfigurationA SkyPilot provider process runs on an Airflow worker, but it stores its metadata into the Airflow master node.
This scheme allows a set of consecutive sky tasks runs across multiple workers by sharing the metadata.Following settings in thedocker-compose.yamldefines the data sharing, including cloud credentials, metadata and workspace.x-airflow-common:environment:volumes:-${AIRFLOW_PROJ_DIR:-.}/dags:/opt/airflow/dags-${AIRFLOW_PROJ_DIR:-.}/logs:/opt/airflow/logs-${AIRFLOW_PROJ_DIR:-.}/config:/opt/airflow/config-${AIRFLOW_PROJ_DIR:-.}/plugins:/opt/airflow/plugins# mount cloud credentials-${HOME}/.aws:/opt/airflow/sky_home_dir/.aws-${HOME}/.azure:/opt/airflow/sky_home_dir/.azure-${HOME}/.config/gcloud:/opt/airflow/sky_home_dir/.config/gcloud-${HOME}/.scp:/opt/airflow/sky_home_dir/.scp# mount sky metadata-${HOME}/.sky:/opt/airflow/sky_home_dir/.sky-${HOME}/.ssh:/opt/airflow/sky_home_dir/.ssh# mount sky working dir-${HOME}/sky_workdir:/opt/airflow/sky_home_dir/sky_workdirThis example mounts the cloud credentials forAWS,Azure,GCP, andSCP,
which have been made by SkyPilot could account setup.
For SkyPilot metadata, check.sky/and.ssh/are placed in your${HOME}directory and mount them.
Additionally, you can mount your own directory likesky_workdir/for user resources including user codes andyamltask definition files for Skypilot execution.Note that all Sky directories are mounted undersky_home_dir/.
They will be symbolic-linked to${HOME}/in workers where a SkyPilot provider process actually runs.UsageThe SkyPilot provider includes the following operators:SkyLaunchOperatorSkyExecOperatorSkyDownOperatorSkySSHOperatorSkyRsyncUpOperatorSkyRsyncDownOperatorSkyLaunchOperatorcreates an cloud cluster and executes a Sky task, as shown below:sky_launch_task=SkyLaunchOperator(task_id="sky_launch_task",sky_task_yaml="~/sky_workdir/my_task.yaml",cloud="cheapest",# aws|azure|gcp|scp|ibm ...gpus="A100:1",minimum_cpus=16,minimum_memory=32,auto_down=False,sky_home_dir='/opt/airflow/sky_home_dir',#set by defaultdag=dag)OnceSkyLaunchOperatorcreates a Sky cluster withauto_down=False, the created cluster can be utilized by the other Sky operators.
Please refer toan example dagfor multiple Sky operators running on a single Sky cluster.
Package: airflow-providers-lokalise
Airflow Provider LokaliseThis repository provides hook and operator to connect to theLokalise APIusing theLokalise Python SDK.ConnectionHook and operator are using the following parameter to connect to Lokalise API:lokalise_conn_id: name of the connection in Airflowpassword: personal API token to connect to the API. Can be obtained followingthis documentationhost: name of the project in Lokalise.Repo organizationHook is located in thelokalise_provider/hooksfolder.Operator is located in thelokalise_provider/operatorfolder.Tests for hook and operator are located in thetestsfolder.DependenciesPython >= 3.10Airflow >= 2.7python-lokalise-api>=2.1.0Additional dependencies are described in thepyproject.toml file.
Package: airflow-providers-oraclethick-hook
airflow-providers-oraclethickProvider for Oracle DB for Airflow 2.X. Just initializes thick client.
Creates connection type OracleThick.Build and install locally:python3 -m build

pip install airflow-providers-oraclethick-hook --no-index --find-links file:///<path>/git/airflow_provider_oracle_thick/dist/Start airflow:airflow webserverCheck if provider is registered and if new connection type has appeared.Install:pip install airflow-providers-oraclethick-hook
Package: airflow-providers-siafi
airflow-providers-siafiProvider para interações com o SIAFI e seus sistemas derivadosInstalaçãopipinstallairflow-providers-siafiConteúdoHook e tipo de conexão "Conta do SIAFI"Usofromairflow.providers.siafi.hooks.siafiimportSIAFIHookwithSIAFIHook('id_conexao')ashook:cpf=hook.cpfsenha=hook.senha...