Package: pytest-airflow
pytest-airflowis a plugin forpytestthat allows tests to be run
within an Airflow DAG.pytesthandles test discovery and function encapsulation, allowing
test declaration to operate in the usual way with the use of
parametrization, fixtures and marks. The generated test callables tests
are eventually passed toPythonOperatorsthat are run as separate
Airflow tasks.Installationpytest-airflowcan be installed withpip:pipinstallpytest-airflowNotepytest-airflowdepends on Apache Airflow, which requiresexport SLUGIFY_USES_TEXT_UNIDECODE=yesto be specified before install. See
theAirflow install instructionsfor background on this requirement.UsageWhen running pytest from the command line, the plugin will collect the
tests and construct the DAG. It will output a DAG tree view in addition to
the requested output.$pytest--airflowWhen invoking pytest from python code,pytest.main()will
return a reference to the DAG.importpytestdag,source,sink=pytest.main(["--airflow","--dag-id","FOO"])The plugin generates two tasks at the start and end of the workflow which
represent the source and sink for the tests. The source task is
responsible for branching and the sink task for reporting. The former and
the later are called__pytest_sourceand__pytest_sinkby default
respectively. In case the user desire to change those defaults name it is
possible to make use of thesourceandsinkflags as below.$pytest--airflow--sourcebranch--sinkreportIf the plugin is installed,pytestwill automatically use it. Saving
the script above in one’s DAG folder is enough to trigger the DAG. Note
thatpytestwill be evaluated from the path where the Airflow
scheduler is invoked.PluginThe plugin creates a DAG of the formsource->tests->sink,sourcemarks tests that will be executed and skipped,testsexecutes the selected tests as separate tasks andsinkreports test
outcome.BranchingAirflow requires that any DAG be completely defined before it is run. So
by the nature of Airflow, we cannot usepytestto collect tests on the
fly based on the results ofsource. Rather,pytestis used to
generate the set of all possible desired tests beforesourceis
evaluated. The user can use all of the available flags topytest(eg.-m,-k, paths) to narrow the set of initial desired tests down.The plugin makes a source task called__pytest_sourceby default
available. This task allows skipping unwanted tests for a particular DAG
run using the following configuration keys:marks: a list of marks, it filters tests in the same way as the-mflag operates when collecting tests withpytest.keywords: a list of keywords, it filters tests in the same way as
the-kflag operates when collecting tests withpytest.FixturesThe plugin defers test execution for the DAG run. That means when callingpytest, the tests will be collected and the associated callables will
be generated and passed to thePythonOperator. If the DAG is compiled
without any errors,pytestwill return the DAG and will exit
sucessfully. That means that it will report that all tests passed, which
only means that the DAG was compiled without any problems.Fixture setup and teardown are executed at the moment of DAG compilation.
That means that fixtures such as database connections will not be
available at the moment of test execution during a DAG run.In order to get around this problem there are two alternatives. The first
alternative is to implement a fixture as a factory, and handling fixture
teardown on the test itself.Alternatively, the plugin allows deferred fixture setup and teardown. In
order to achieve deferred execution, the name of the fixture must be
prefixed withdefer_or it must depend on the reserved fixturetask_ctx. That means that the plugin defer the execution of such
fixtures until the DAG is run. Fixtures that depend on a deferred fixture
will also have its execution deferred for later.The reserved fixturetask_ctxis always deferred. This fixture
evaluates the Airflow task context and is available to the user when
writting tests. Using this fixture, the user has access to all the items
that would be available tokwargswhen settingprovide_contexttoTruewhen using thePythonOperatorin Airflow.All in all, collection time fixture execution should be used for test
parametrization, for generating expensive resources that can be made
available to tests as copies and for generating fixture factories. On the
other hand, deferred fixtures are great for database connections and other
resources that need to be recycled at each test execution.ReportingFinally, the sink taskreportcan be used for reporting purposes and for
communicating test results to other DAGs using thexcomchannel.  The user
can supply its owndag_reportfixture for customizing its reporting
requirements. The plugin expects the following fixture signature, scoped at thesessionlevel.@pytest.fixture(scope="session")defdag_report(**kwargs):...DAG ConfigurationThe user can configure the DAG using two reserved fixtures for this. The
fixtures must be scoped at thesessionlevel and its location should cover
all the collected test items. The most narrow fixture that covers all of the
collected items will be selected. Otherwise, the plugin uses default values for
those fixtures. Apart from that, fixture execution and discovery should operate
in the usual way.The first fixture isdag_default_args, which should return
a dictionary withdefault_argsthat will be passed to the dag
initialization. The default returns{"owner":"airflow","start_date":datetime.datetime(2018,1,1),"end_date":None,"depends_on_past":False,}The second fixture isdagwhich should return an Airflow DAG that will
be used throughout the script.If the user desires only to modify the name of the DAG, it is possible to
simply pass the--dag-idflag to thepytestcmdline.If the user desires to integrate the DAG generated from this plugin in
her/his own DAG. One option is to define the whole DAG inside the sameconftest.pyfile that is used bypytestto initialize the tests.
If this is not possible and the DAG must be defined separately, it is
possible to create a custompytestplugin in the same file where the
DAG is created and pass such plugin topytest.mainas the example
below illustrates.importpytestfromairflowimportDAGmy_dag=DAG(dag_id="foo",start_date="2017-01-01")classMyPlugin:@pytest.fixture(scope="session")defdag(self):returnmy_dagmy_dag,source,sink=pytest.main(["--airflow"],plugins=[MyPlugin()])LicenseThis Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, You can obtain one athttp://mozilla.org/MPL/2.0/.