Package: airflow-provider-sample
Airflow Sample ProviderGuidelines on building, deploying, and maintaining provider packages that will help Airflow users interface with external systems. Maintained with ❤️ by Astronomer.This repository provides best practices for building, structuring, and deploying Airflow provider packages as independent python modules available on PyPI.Provider repositories must be public on Github and follow the structural and technical guidelines laid out in this Readme. Ensure that all of these requirements have been met before submitting a provider package for community review.Here, you'll find information on requirements and best practices for key aspects of your project:File formattingDevelopmentAirflow integrationDocumentationTestingFormatting StandardsBefore writing and testing the functionality of your provider package, ensure that your project follows these formatting conventions.Package nameThe highest level directory in the provider package should be named in the following format:airflow-provider-<provider-name>Repository structureAll provider packages must adhere to the following file structure:├──LICENSE# A license is required, MIT or Apache is preferred.├──README.md
├──sample_provider# Your package import directory. This will contain all Airflow modules and example DAGs.│├──__init__.py
│├──example_dags
││├──__init__.py
││└──sample-dag.py
│├──hooks
││├──__init__.py
││└──sample_hook.py
│├──operators
││├──__init__.py
││└──sample_operator.py
│└──sensors
│├──__init__.py
│└──sample_sensor.py
├──setup.py# A setup.py file to define dependencies and how the package is built and shipped. If you'd like to use setup.cfg, that is fine as well.└──tests# Unit tests for each module.├──__init__.py├──hooks│├──__init__.py│└──sample_hook_test.py├──operators│├──__init__.py│└──sample_operator_test.py└──sensors├──__init__.py└──sample_sensor_test.pyDevelopment StandardsIf you followed the formatting guidelines above, you're now ready to start editing files to include standard package functionality.Python Packaging ScriptsYoursetup.pyfile should contain all of the appropriate metadata and dependencies required to build your package. Use thesamplesetup.pyfilein this repository as a starting point for your own project.If some of the options for building your package are variables or user-defined, you can specify asetup.cfgfile instead.Managing DependenciesWhen building providers, these guidelines will help you avoid potential for dependency conflicts:It is important that the providers do not include dependencies that conflict with the underlying dependencies for a particular Airflow version. All of the default dependencies included in the core Airflow project can be found in the Airflowsetup.py file.Keep all dependencies relaxed at the upper bound. At the lower bound, specify minor versions (for example,depx >=2.0.0, <3).VersioningUse standard semantic versioning for releasing your package. When cutting a new release, be sure to update all of the relevant metadata fields in your setup file.Building ModulesAll modules must follow a specific set of best practices to optimize their performance with Airflow:All classes should always be able to run without access to the internet.The Airflow Scheduler parses DAGs on a regular schedule. Every time that parse happens, Airflow will execute whatever is contained in theinitmethod of your class. If thatinitmethod contains network requests, such as calls to a third party API, there will be problems due to repeated network calls.Init methods should never call functions which return valid objects only at runtime. This will cause a fatal import error when trying to import a module into a DAG. A common best practice for referencing connectors and variables within DAGs is to useJinja Templating.All operator modules need anexecutemethod.This method defines the logic that the operator will implement.Modules should also take advantage of native Airflow features that allow your provider to:Register custom connection types, which improve the user experience when connecting to your tool.Includeextra-linksthat link your provider back to its page on the Astronomer Registry. This provides users easy access to documentation and example DAGs.Refer to theAirflow Integration Standardssection for more information on how to build in these extra features.Unit testingYour top-leveltests/folder should include unit tests for all modules that exist in the repository. You can write tests in the framework of your choice, but the Astronomer team and Airflow community typically usepytest.You can test this package by running:python3 -m unittestfrom the top-level of the directory.Airflow Integration StandardsAirflow exposes a number of plugins to interface from your provider package. We highly encourage provider maintainers to add these plugins because they significantly improve the user experience when connecting to a provider.Defining an entrypointTo enable custom connections, you first need to define anapache_airflow_providerentrypoint in yoursetup.pyorsetup.cfgfile:entry_points={
  "apache_airflow_provider": [
      "provider_info=sample_provider.__init__:get_provider_info"
        ]
    }Next, you need to add aget_provider_infomethod to the__init__file in your top-level provider folder. This function needs to return certain metadata associated with your package in order for Airflow to use it at runtime:defget_provider_info():return{"package-name":"airflow-provider-sample'","name":"Sample Airflow Provider",# Required"description":"A sample template for airflow providers.",# Required"hook-class-names":["sample_provider.hooks.sample_hook.SampleHook"],"extra-links":["sample_provider.operators.sample_operator.ExtraLink"],"versions":["0.0.1"]# Required}Once you define the entrypoint, you can use native Airflow features to expose custom connection types in the Airflow UI, as well as additional links to relevant documentation.Adding Custom Connection FormsAirflow enables custom connection forms through discoverable hooks. The following is an example of a custom connection form for the Fivetran provider:Add code to the hook class to initiate a discoverable hook and create a custom connection form. The following code defines a hook and a custom connection form:classExampleHook(BaseHook):"""ExampleHook docstring..."""conn_name_attr='example_conn_id'default_conn_name='example_default'conn_type='example'hook_name='Example'@staticmethoddefget_connection_form_widgets()->Dict[str,Any]:"""Returns connection widgets to add to connection form"""fromflask_appbuilder.fieldwidgetsimportBS3PasswordFieldWidget,BS3TextFieldWidgetfromflask_babelimportlazy_gettextfromwtformsimportPasswordField,StringField,BooleanFieldreturn{"extra__example__bool":BooleanField(lazy_gettext('Example Boolean')),"extra__example__account":StringField(lazy_gettext('Account'),widget=BS3TextFieldWidget()),"extra__example__secret_key":PasswordField(lazy_gettext('Secret Key'),widget=BS3PasswordFieldWidget()),}@staticmethoddefget_ui_field_behaviour()->Dict:"""Returns custom field behaviour"""importjsonreturn{"hidden_fields":['port'],"relabeling":{},"placeholders":{'extra':json.dumps({"example_parameter":"parameter",},indent=1,),'host':'example hostname','schema':'example schema','login':'example username','password':'example password','extra__example__account':'example account name','extra__example__secret_key':'example secret key',},}Some notes about using custom connections:get_connection_form_widgets()creates extra fields using flask_appbuilder. Extra fields are defined in the following format:extra__<conn_type>__<field_name>A variety of field types can be created using this function, such as strings, passwords, booleans, and integers.get_ui_field_behaviour()is a JSON schema describing the form field behavior. Fields can be hidden, relabeled, and given placeholder values.To connect a form to Airflow, add the hook class name of a discoverable hook to"hook-class-names"in theget_provider_infomethod as mentioned inDefining an entrypoint.Adding Custom LinksOperators can add custom links that users can click to reach an external source when interacting with an operator in the Airflow UI. This link can be created dynamically based on the context of the operator. The following code example shows how to initiate an extra link within an operator:classExampleLink(BaseOperatorLink):"""Link for ExmpleOperator"""name='Example Link'defget_link(self,operator,dttm):"""Get link to registry page."""registry_link="https://{example}.com"returnregistry_link.format(example='example')classExampleOperator(BaseOperator):"""ExampleOperator docstring..."""operator_extra_links=(ExampleLink(),)To connect custom links to Airflow, add the operator class name to"extra-links"in theget_provider_infomethod mentioned above.Documentation StandardsCreating excellent documentation is essential for explaining the purpose of your provider package and how to use it.Inline Module DocumentationEvery Python module, including all hooks, operators, sensors, and transfers, should be documented inline viasphinx-templated docstrings. These docstrings should be included at the top of each module file and contain three sections separated by blank lines:A one-sentence description explaining what the module does.A longer description explaining how the module works. This can include details such as code blocks or blockquotes. For more information Sphinx markdown directives, read theSphinx documentation.A declarative definition of parameters that you can pass to the module, templated per the example below.For a full example of inline module documentation, see theexample operator in this repository.READMEThe README for your provider package should give users an overview of what your provider package does. Specifically, it should include:High-level documentation about the provider's service.Steps for building a connection to the service from Airflow.What modules exist within the package.An exact set of dependencies and versions that your provider has been tested with.Guidance for contributing to the provider package.Functional Testing StandardsTo build your repo into a python wheel that can be tested, follow the steps below:Clone the provider repo.cdinto provider directory.Runpython3 -m pip install build.Runpython3 -m buildto build the wheel.Find the .whl file in/dist/*.whl.Download theAstro CLI.Create a new project directory, cd into it, and runastro dev initto initialize a new astro project.Ensure the Dockerfile contains the Airflow 2.0 image:FROM quay.io/astronomer/ap-airflow:2.0.0-buster-onbuildCopy the.whlfile to the top level of your project directory.Install.whlin your containerized environment by adding the following to your Dockerfile:RUN pip install --user airflow_provider_<PROVIDER_NAME>-0.0.1-py3-none-any.whlCopy your sample DAG to thedags/folder of your astro project directory.Runastro dev startto build the containers and run Airflow locally (you'll need Docker on your machine).When you're done, runastro dev stopto wind down the deployment. Runastro dev killto kill the containers and remove the local Docker volume. You can also useastro dev killto stop the environment before rebuilding with a new.whlfile.Note: If you are having trouble accessing the Airflow webserver locally, there could be a bug in your wheel setup. To debug, rundocker ps, grab the container ID of the scheduler, and rundocker logs <scheduler-container-id>to inspect the logs.Once you have built and tested your provider package as a Python wheel, you're ready tosend us your repoto be published onThe Astronomer Registry.