Package: astronomer-providers
WarningThe majority of operators and sensors within this repository have been deprecated and will not receive further updates.
Read more about the deprecation in theDeprecation Noticesection below.Deprecation NoticeWith the release1.19.0of the astronomer-providers package, most of the operators and sensors are deprecated and will
no longer receive updates. We recommend migrating to the official Apache Airflow Providers for the latest features and support.
For the operators and sensors that are deprecated in this repository, migrating to the official Apache Airflow Providers
is as simple as changing the import path fromfrom astronomer.providers.*.*.operator_module import SomeOperatorAsynctofrom airflow.providers.*.*.operator_module import SomeOperatorand setting thedeferrableargument toTruewhile using the operator or sensor in your DAG.
Setting thedeferrableargument toTruewill ensure that the operator or sensor is using the async version
of the operator or sensor from the official Apache Airflow Providers.For example, to migrate fromastronomer.providers.amazon.aws.operators.batch.BatchOperatorAsynctoairflow.providers.amazon.aws.operators.s3.BatchOperator, simply change the import path and pass
the deferrable argument:BatchOperator(task_id="copy_object",your_arguments,your_keyword_arguments,deferrable=True,)For more information on using the deferrable operators and sensors from the official Apache Airflow Providers, visit the following links:https://airflow.apache.org/docs/apache-airflow-providers/index.htmlhttps://airflow.apache.org/docs/#providers-packages-docs-apache-airflow-providers-index-htmlhttps://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.htmlNoteAlthough the default value for thedeferrableargument isFalse, it’s possible to configure the default value for thedeferrableargument across
your deployment by setting thedefault_deferrableflag in theoperatorssections of your Airflow configuration. Once you set thedefault_deferrableflag toTrue,
you can remove thedeferrableargument from your operators and sensors and they will use the async version of the operator or sensor from the official Apache Airflow Providers
if it exists.See more at:https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#default-deferrableFor troubleshooting of issues with migrations, you are suggested to open up aGitHub discussionInstallationInstall and update usingpip:pipinstallastronomer-providersThis only installs dependencies for core provider. To install all dependencies, run:pipinstall'astronomer-providers[all]'To only install the dependencies for a specific provider, specify the integration name as extra argument, example
to install Kubernetes provider dependencies, run:pipinstall'astronomer-providers[cncf.kubernetes]'ExtrasExtra NameInstallation CommandDependenciesallpip install'astronomer-providers[all]'Allamazonpip install'astronomer-providers[amazon]'Amazonapache.hivepip install'astronomer-providers[apache.hive]'Apache Hiveapache.livypip install'astronomer-providers[apache.livy]'Apache Livycncf.kubernetespip install'astronomer-providers[cncf.kubernetes]'Cncf Kubernetesdatabrickspip install'astronomer-providers[databricks]'Databricksdbt.cloudpip install'astronomer-providers[dbt.cloud]'Dbt Cloudgooglepip install'astronomer-providers[google]'Googlehttppip install'astronomer-providers[http]'Httpmicrosoft.azurepip install'astronomer-providers[microsoft.azure]'Microsoft Azureopenlineagepip install'astronomer-providers[openlineage]'Openlineagesftppip install'astronomer-providers[sftp]'Sftpsnowflakepip install'astronomer-providers[snowflake]'SnowflakeExample UsageThis repo is structured same as the Apache Airflow’s source code, so for example
if you want to import Async operators, you can import it as follows:fromastronomer.providers.amazon.aws.sensors.s3importS3KeySensorAsyncasS3KeySensorwaiting_for_s3_key=S3KeySensor(task_id="waiting_for_s3_key",bucket_key="sample_key.txt",wildcard_match=False,bucket_name="sample-bucket",)Example DAGsfor each provider is within the respective provider’s folder. For example,
the Kubernetes provider’s DAGs are within theastronomer/providers/cncf/kubernetes/example_dagsfolder.PrincipleWe will only create Async operators for the “sync-version” of operators that do some level of polling
(take more than a few seconds to complete).For example, we won’t create an async Operator for aBigQueryCreateEmptyTableOperatorbut will create one
forBigQueryInsertJobOperatorthat actually runs queries and can take hours in the worst case for task completion.To create async operators, we need to inherit from the corresponding airflow sync operators.
If sync version isn’t available, then inherit from airflowBaseOperator.To create async sensors, we need to inherit from the corresponding sync sensors.
If sync version isn’t available, then inherit from airflowBaseSensorOperator.ChangelogWe followSemantic Versioningfor releases.
CheckCHANGELOG.rstfor the latest changes.Contributing GuideAll contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.A detailed overview on how to contribute can be found in theContributing Guide.As contributors and maintainers to this project, you are expected to abide by theContributor Code of Conduct.Goals for the projectOur focus is on the speed of iteration and development in this stage of the project and so we want to be able to
quickly iterate with our community members and customers and cut releases as necessaryAirflow Providers are separate packages from the coreapache-airflowpackage and we would like to avoid
further bloating the Airflow repoWe want users and the community to be able to easily track features and the roadmap for individual providers
that we developWe would love to see the Airflow community members create, maintain and share their providers to build an Ecosystem
of Providers.LimitationsIn Airflow sensors have a parammodewhich can bepokeandreschedule.
In async sensors, this param has no usage since tasks gets deferred to Triggerer.LicenseApache License 2.0