Saravanan666/pypi-org huggingface dataset:
Package: apache-airflow-granulate-databricks
Granulate Airflow-Databricks IntegrationOverviewThe Granulate Airflow-Databricks Integration is an open-source plugin for Apache Airflow. It's specifically designed to set environment variables that allow Granulate's performance monitoring agent to identify and integrate with Databricks jobs orchestrated by Airflow. This plugin tags Databricks jobs, aiding the Granulate optimizing agent.Modes of OperationThe Granulate plugin can operate in three different modes:Passive Mode: In this mode, you need to manually replaceDatabricksSubmitRunOperatorandDatabricksSubmitRunDeferrableOperatorwithGranulateDatabricksSubmitRunOperatorandGranulateDatabricksSubmitRunDeferrableOperatorrespectively in your DAGs. This mode allows you to selectively apply Granulate to specific operators. TheGranulateoperators are drop-in replacement for their respective operators, so you can just swap them in code for relevant DAGs.Auto-Patch for Specific DAGs: To enable auto-patching on specific DAGs, import and invoke thepatchfunction from the plugin at the beginning of your DAG file:fromapache_airflow_granulate_databricks.granulate_pluginimportpatchpatch()This method patches the Databricks operators in the DAG where it's called, enabling the Granulate environment variable.Auto-Patch for All DAGs: For an all-encompassing approach, install the plugin with the 'auto-patch' extra:pip install apache-airflow-granulate-databricks[auto-patch]This mode patches all Databricks operators across all DAGs in your Airflow environment, automatically applying Granulate enhancements.InstallationTo install the apache-airflow-granulate-databricks package, choose the method that best fits your setup:Using pip:For a standard installation, run:pip install apache-airflow-granulate-databricksTo enable automatic DAG patching, include the auto-patch extra:pip install apache-airflow-granulate-databricks[auto-patch]Using_PIP_ADDITIONAL_REQUIREMENTSin Airflow:Append the following line to your _PIP_ADDITIONAL_REQUIREMENTS:apache-airflow-granulate-databricksFor auto-patching, use:apache-airflow-granulate-databricks[auto-patch]Restart your Airflow services to apply these changes.Package RemovalIf you used pip to install, run:pip uninstall apache-airflow-granulate-databricksIf you used_PIP_ADDITIONAL_REQUIREMENTS, removeapache-airflow-granulate-databricksMake sure to revert your code if you used Granulate's operators, or you you used thepatchfunction.Restart your Airflow services to apply these changes.RequirementsTested on Databricks Airflow Provider (4.2.0 <= version <= 6.0.0)Python version 3.7 or higherLicenseThis project is licensed under the Apache License 2.0 - see theLICENSEfile for details.SupportFor support, questions, or issues, please open an issue in the GitHub repository.
Package: apache-airflow-providers-aerofoil
# Aerofoil : Lifting Airflow
Aerofoil is an Apache Airflow plugin. It provides some very useful extensions, that are missing in the Airflow. The plugin is available with MIT license, the intention is to keep it simple and friendly.### Features:
##### Backfill UI:
A convenient way to run Backfill DAG from UI. In most of the enterprise setup, DAG developers do not have access to Airflow servers to be able to run backfill jobs. This makes running backfill an expensive operation. With the Backfilll UI, Airflow users can submit backfill command from UI. The backfill itself runs as a Airflow DAG and is compatible with most production Airflow configurations. This is also the most used feature in Aerofoil.
##### My DAGs
Large Airflow installations can have hundreds of DAGs. This feature allows logged in user to view to see the DAG’s authored by her/him. The DAG Author must be set correctly in the DAG file and it must match with the Airflow user name.
##### Reset DAG History
Sometime complete DAG history needs to be reset. There is currently no way of clearing a DAGs history in Airflow. This functionality allows clearing/resetting DAGs history via a convenient UI.##### Fake Sucess
A shortcut to mark your DAG/Task runs successful. The Fake success is a powerful featurs, and to avoid misuse and Audit Trail is maintained. Adminstratos can always find out, who is trying to fake it.##### AerofoilBashOperator:
Context aware BashOperator, that can take a context and pass it to callback e.g.on_success_callback,on_failure_callbacketc.## Installation
### Pre-requisite
- Working Airflow installation.
- works with any executor other than Sequential Executor or LocalExecutor (e.g. KubernetasExecutor, CeleryExecutor)
- Requires a non-sqlite database in the backend.### Install Instructions:
- pip install apache-airflow-providers-aerofoil
- Add rquired tables to Airflow Metadata DB by executing src/aerofoil/models.sql
- Deploy DAG file in the src/aerofoil/aerofoil_backfill.py DAGs to  your Airflow (usually by putting the DAG in  Airflow’s DAGs folder.
- Make sure the__aerofoil_backfill__- DAG is enabled.## Backfilll Design:
When the user submits a backfill, it creates an entry in theaerofoil_backfillbackfill table.__aerofoil_backfill__DAG is scheduled to run every minute (which can be changed, by modifying the DAG). The Sesor in the DAG, picks up the entries in the DB and create dynamic task for each entry. The Backfill job itself runs as a Bash command in the executor. This provides a flexible and executor agnostic design.## Screenshots:
##### Aerofoil Menu
![img](screenshots/menu.png)
##### Backfill Screen
![img](screenshots/backfill.png)
##### Reset DAG Menu
![img](screenshots/reset.png)
##### Fake Success Menu
![img](screenshots/fake_success.png)## Contribute:
If you find something missing and would like to contrinute, feel free to raise a pull request. If you find a bug and would like me to fix it, please feel free to raise an issue.
Package: apache-airflow-providers-aerofoil2
# Aerofoil : Lifting Airflow
Aerofoil is an Apache Airflow plugin. It provides some very useful extensions, that are missing in the Airflow. The plugin is available with MIT license, the intention is to keep it simple and friendly.### Features:
##### Backfill UI:
A convenient way to run Backfill DAG from UI. In most of the enterprise setup, DAG developers do not have access to Airflow servers to be able to run backfill jobs. This makes running backfill an expensive operation. With the Backfilll UI, Airflow users can submit backfill command from UI. The backfill itself runs as a Airflow DAG and is compatible with most production Airflow configurations. This is also the most used feature in Aerofoil.
##### My DAGs
Large Airflow installations can have hundreds of DAGs. This feature allows logged in user to view to see the DAG’s authored by her/him. The DAG Author must be set correctly in the DAG file and it must match with the Airflow user name.
##### Reset DAG History
Sometime complete DAG history needs to be reset. There is currently no way of clearing a DAGs history in Airflow. This functionality allows clearing/resetting DAGs history via a convenient UI.##### Fake Sucess
A shortcut to mark your DAG/Task runs successful. The Fake success is a powerful featurs, and to avoid misuse and Audit Trail is maintained. Adminstratos can always find out, who is trying to fake it.##### AerofoilBashOperator:
Context aware BashOperator, that can take a context and pass it to callback e.g.on_success_callback,on_failure_callbacketc.## Installation
### Pre-requisite
- Working Airflow installation.
- works with any executor other than Sequential Executor or LocalExecutor (e.g. KubernetasExecutor, CeleryExecutor)
- Requires a non-sqlite database in the backend.### Install Instructions:
- pip install apache-airflow-providers-aerofoil
- Add rquired tables to Airflow Metadata DB by executing src/aerofoil/models.sql
- Deploy DAG file in the src/aerofoil/aerofoil_backfill.py DAGs to  your Airflow (usually by putting the DAG in  Airflow’s DAGs folder.
- Make sure the__aerofoil_backfill__- DAG is enabled.## Backfilll Design:
When the user submits a backfill, it creates an entry in theaerofoil_backfillbackfill table.__aerofoil_backfill__DAG is scheduled to run every minute (which can be changed, by modifying the DAG). The Sesor in the DAG, picks up the entries in the DB and create dynamic task for each entry. The Backfill job itself runs as a Bash command in the executor. This provides a flexible and executor agnostic design.## Screenshots:
##### Aerofoil Menu
![img](screenshots/menu.png)
##### Backfill Screen
![img](screenshots/backfill.png)
##### Reset DAG Menu
![img](screenshots/reset.png)
##### Fake Success Menu
![img](screenshots/fake_success.png)## Contribute:
If you find something missing and would like to contrinute, feel free to raise a pull request. If you find a bug and would like me to fix it, please feel free to raise an issue.
Package: apache-airflow-providers-airbyte
Packageapache-airflow-providers-airbyteRelease:3.6.0AirbyteProvider packageThis is a provider package forairbyteprovider. All classes for this provider package
are inairflow.providers.airbytepython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-airbyteThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0apache-airflow-providers-httpCross provider package dependenciesThose are dependencies that might be needed in order to use all the features of the package.
You need to install the specified provider packages in order to use them.You can install such cross-provider dependencies when installing from PyPI. For example:pipinstallapache-airflow-providers-airbyte[http]Dependent packageExtraapache-airflow-providers-httphttpThe changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-alibaba
Packageapache-airflow-providers-alibabaRelease:2.7.2Alibaba Cloud integration (includingAlibaba Cloud).Provider packageThis is a provider package foralibabaprovider. All classes for this provider package
are inairflow.providers.alibabapython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-alibabaThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0oss2>=2.14.0alibabacloud_adb20211201>=1.0.0alibabacloud_tea_openapi>=0.3.7The changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-amazon
Packageapache-airflow-providers-amazonRelease:8.18.0Amazon integration (includingAmazon Web Services (AWS)).Provider packageThis is a provider package foramazonprovider. All classes for this provider package
are inairflow.providers.amazonpython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-amazonThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0apache-airflow-providers-common-sql>=1.3.1apache-airflow-providers-httpboto3>=1.33.0botocore>=1.33.0inflection>=0.5.1watchtower>=2.0.1,<4jsonpath_ng>=1.5.3redshift_connector>=2.0.918sqlalchemy_redshift>=0.8.6asgirefPyAthena>=3.0.10Cross provider package dependenciesThose are dependencies that might be needed in order to use all the features of the package.
You need to install the specified provider packages in order to use them.You can install such cross-provider dependencies when installing from PyPI. For example:pipinstallapache-airflow-providers-amazon[apache.hive]Dependent packageExtraapache-airflow-providers-apache-hiveapache.hiveapache-airflow-providers-cncf-kubernetescncf.kubernetesapache-airflow-providers-common-sqlcommon.sqlapache-airflow-providers-exasolexasolapache-airflow-providers-ftpftpapache-airflow-providers-googlegoogleapache-airflow-providers-httphttpapache-airflow-providers-imapimapapache-airflow-providers-microsoft-azuremicrosoft.azureapache-airflow-providers-mongomongoapache-airflow-providers-openlineageopenlineageapache-airflow-providers-salesforcesalesforceapache-airflow-providers-sshsshThe changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-anomalo
Apache Airflow Provider for AnomaloA set of native Airflow operators forAnomaloCompatibilityThese operators were created and tested withPython 3.8-3.10Airflow 2.3+Anomalo python client 0.0.7Installationpip install apache-airflow-providers-anomaloYou can validate that it is correctly installed by runningairflow providers liston the command line and seeing ifapache-airflow-providers-anomalois a listed providers package.Airflow SetupFrom the airflow UI, go to Admin > Connections and hit the+button at the top to add a new connection.From the "Connection Type" drop down, select "Anomalo".Then fill in the fields for "Connection Id" (anomalo-defaultis the default connection id), "Host", and "API Secret Token".UsageObtain Anomalo table name from GUI. For examplewould bepublic-bq.covid19_nyt.us_countiesThis package includes 3 different operators. You can find documentation for them on the operator code itself.Run checks Operator:airflow.providers.anomalo.operators.anomalo.AnomaloRunCheckOperatorJob Sensorairflow.providers.anomalo.sensors.anomalo.AnomaloJobCompleteSensorValidate table checks:airflow.providers.anomalo.operators.anomalo.AnomaloPassFailOperatorSeeanomalo_dag_example.pyfor usage exampleReleasing to PyPiTo release a new version to PyPi, you will need toConfigure your pypi token by running:poetry config pypi-token.pypi [token here]The token can be found in the AWS secrets managerBump the version number in pyproject.toml. Make sure that this change is committed.runpoetry publish --build
Package: apache-airflow-providers-apache-beam
Packageapache-airflow-providers-apache-beamRelease:5.6.1Apache Beam.Provider packageThis is a provider package forapache.beamprovider. All classes for this provider package
are inairflow.providers.apache.beampython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-apache-beamThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0apache-beam>=2.53.0pyarrow>=14.0.1Cross provider package dependenciesThose are dependencies that might be needed in order to use all the features of the package.
You need to install the specified provider packages in order to use them.You can install such cross-provider dependencies when installing from PyPI. For example:pipinstallapache-airflow-providers-apache-beam[google]Dependent packageExtraapache-airflow-providers-googlegoogleThe changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-apache-cassandra
Packageapache-airflow-providers-apache-cassandraRelease:3.4.1Apache Cassandra.Provider packageThis is a provider package forapache.cassandraprovider. All classes for this provider package
are inairflow.providers.apache.cassandrapython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-apache-cassandraThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0cassandra-driver>=3.13.0The changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-apache-drill
Packageapache-airflow-providers-apache-drillRelease:2.6.1Apache Drill.Provider packageThis is a provider package forapache.drillprovider. All classes for this provider package
are inairflow.providers.apache.drillpython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-apache-drillThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0apache-airflow-providers-common-sql>=1.3.1sqlalchemy-drill>=1.1.0Cross provider package dependenciesThose are dependencies that might be needed in order to use all the features of the package.
You need to install the specified provider packages in order to use them.You can install such cross-provider dependencies when installing from PyPI. For example:pipinstallapache-airflow-providers-apache-drill[common.sql]Dependent packageExtraapache-airflow-providers-common-sqlcommon.sqlThe changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-apache-druid
Packageapache-airflow-providers-apache-druidRelease:3.8.1Apache Druid.Provider packageThis is a provider package forapache.druidprovider. All classes for this provider package
are inairflow.providers.apache.druidpython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-apache-druidThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0apache-airflow-providers-common-sql>=1.3.1pydruid>=0.4.1Cross provider package dependenciesThose are dependencies that might be needed in order to use all the features of the package.
You need to install the specified provider packages in order to use them.You can install such cross-provider dependencies when installing from PyPI. For example:pipinstallapache-airflow-providers-apache-druid[apache.hive]Dependent packageExtraapache-airflow-providers-apache-hiveapache.hiveapache-airflow-providers-common-sqlcommon.sqlThe changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-apache-flink
Packageapache-airflow-providers-apache-flinkRelease:1.3.0Apache FlinkProvider packageThis is a provider package forapache.flinkprovider. All classes for this provider package
are inairflow.providers.apache.flinkpython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-apache-flinkThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0cryptography>=2.0.0apache-airflow-providers-cncf-kubernetes>=5.1.0Cross provider package dependenciesThose are dependencies that might be needed in order to use all the features of the package.
You need to install the specified provider packages in order to use them.You can install such cross-provider dependencies when installing from PyPI. For example:pipinstallapache-airflow-providers-apache-flink[cncf.kubernetes]Dependent packageExtraapache-airflow-providers-cncf-kubernetescncf.kubernetesThe changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-apache-hdfs
Packageapache-airflow-providers-apache-hdfsRelease:4.3.2Hadoop Distributed File System (HDFS)andWebHDFS.Provider packageThis is a provider package forapache.hdfsprovider. All classes for this provider package
are inairflow.providers.apache.hdfspython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-apache-hdfsThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0hdfs[avro,dataframe,kerberos]>=2.0.4The changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-apache-hive
Packageapache-airflow-providers-apache-hiveRelease:7.0.0Apache HiveProvider packageThis is a provider package forapache.hiveprovider. All classes for this provider package
are inairflow.providers.apache.hivepython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-apache-hiveThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0apache-airflow-providers-common-sql>=1.3.1hmsclient>=0.1.0pandas>=1.2.5pyhive[hive_pure_sasl]>=0.7.0thrift>=0.9.2Cross provider package dependenciesThose are dependencies that might be needed in order to use all the features of the package.
You need to install the specified provider packages in order to use them.You can install such cross-provider dependencies when installing from PyPI. For example:pipinstallapache-airflow-providers-apache-hive[amazon]Dependent packageExtraapache-airflow-providers-amazonamazonapache-airflow-providers-common-sqlcommon.sqlapache-airflow-providers-microsoft-mssqlmicrosoft.mssqlapache-airflow-providers-mysqlmysqlapache-airflow-providers-prestoprestoapache-airflow-providers-sambasambaapache-airflow-providers-verticaverticaThe changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-apache-impala
Packageapache-airflow-providers-apache-impalaRelease:1.3.0Apache Impala.Provider packageThis is a provider package forapache.impalaprovider. All classes for this provider package
are inairflow.providers.apache.impalapython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-apache-impalaThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredimpyla>=0.18.0,<1.0apache-airflow>=2.6.0Cross provider package dependenciesThose are dependencies that might be needed in order to use all the features of the package.
You need to install the specified provider packages in order to use them.You can install such cross-provider dependencies when installing from PyPI. For example:pipinstallapache-airflow-providers-apache-impala[common.sql]Dependent packageExtraapache-airflow-providers-common-sqlcommon.sqlThe changelog for the provider package can be found in thechangelog.
Package: apache-airflow-providers-apache-kafka
Packageapache-airflow-providers-apache-kafkaRelease:1.3.1Apache KafkaProvider packageThis is a provider package forapache.kafkaprovider. All classes for this provider package
are inairflow.providers.apache.kafkapython package.You can find package information and changelog for the provider
in thedocumentation.InstallationYou can install this package on top of an existing Airflow 2 installation (seeRequirementsbelow
for the minimum Airflow version supported) viapip installapache-airflow-providers-apache-kafkaThe package supports the following python versions: 3.8,3.9,3.10,3.11RequirementsPIP packageVersion requiredapache-airflow>=2.6.0asgirefconfluent-kafka>=1.8.2The changelog for the provider package can be found in thechangelog.