Package: airflow-pydantic-dags
Airflow-Pydantic-DAGsPydanticDAGs allows you to usePydanticmodels for task configuration in yourAirflowDAGs.Airflow + Pydantic = ❤️: Use Pydantic for Airflow Task ConfigurationRuns of Airflow DAGs can be configured using parameters, that usepropietary validationand are not de-serialized: you get a dictionary of parameters in your tasks. This leaves YOU to deal with un-typed dictionary values at the task level, write validation logic for them, all without instrospection when developing.If only there was a established library to create data models with validation?EnterPydantic: using theDAGmodel ofairflow-pydantic-dagsyou get a Pydantic model passed to your tasks, that contains your validated run configuration. Your model is also exposed in the Airflow UI, making it easy to launch Airflow DAGRuns.InstallationCurrently, this supports:airflow >= 2.6.0pydantic < 2As soon as we publish this to pip :) you canpip install airflow-pydantic-dagsUsageUse the classPydanticDAGinstead ofDAG, and pass the pydantic class you want to use
to parse the params. By decorating tasks with@dag.parse_config()you will get aconfig_objectpassed to your task, which is an instance of the Pydantic class, initialized with your parameters.Currently, your Pydantic classneeds to provide default values for all attributes,
otherwise the DAG will fail to initialize.In the Airflow UI, you will find all attributes of the Pydantic class exposed as
params. Currently, only non-nested fields are exposed as single items, everything
else will become a json parameter.[!NOTE]
Validation of params by Pydantic when submitting through the UI and CLI iscurrentlynot
done at time of DAG run creation, but instead only when the parameters are first accessed
in a task. Achieving validation at creation time will be part of a future update.
See just below for how to force earlier validation.To achieve systematic early validation and fail your PydanticDAGdagfor invalid parameters, do one of the following:usePydanticDAG(add_validation_task=True): this will add a task
(without dependencies) to your DAG that validates the paramsusedag.get_validation_taskto get a task that validates the params:
You can use this to create custom dependencies in your DAG on the validationuse the decorator@dag.parse_configon any of your own tasks:
this will force validation of the paramsExampleSource is atdags/example.py:fromairflow_pydantic_dags.dagimportPydanticDAGfromairflow.decoratorsimporttaskfromdatetimeimportdatetimeclassMyRunConfig(pyd.BaseModel):string_to_print:str="overwrite this"withPydanticDAG(params_pydantic_class=MyRunConfig,dag_id="example",schedule=None,start_date=datetime(2023,8,1),params={"airflow_classic_param":1},)asdag:@task(dag=dag)@dag.parse_config()defpull_params(config_object:MyRunConfig|None=None,**kwargs):# params contains pydantic and non-pydantic parameter valuesprint("Params:")print(kwargs["params"])# using the dag.parse_config() decorator, we also get the deserialized pydantic object as 'config_object'print("Pydantic object:")print(type(config_object))print(config_object)pull_params()This generates an UI interface for your DAG, including all pydantic and non-pydantic parameters:And the task log shows{logging_mixin.py:151} INFO - Params:
{logging_mixin.py:151} INFO - {'airflow_classic_param': 1, 'string_to_print': 'overwrite this'}
{logging_mixin.py:151} INFO - Pydantic object:
{logging_mixin.py:151} INFO - <class 'unusual_prefix_95fa66c061cb15347627f327a8a577346657e3a7_example.MyRunConfig'>
{logging_mixin.py:151} INFO - string_to_print='overwrite this'MentionsOther projectspyproject.toml,setup.cfg, and.pre-commit-config.yamlwere adapted from the excellentcoookiecutter-djangoproject.