Package: openlineage-airflow
OpenLineage Airflow IntegrationA library that integratesAirflowDAGswithOpenLineagefor automatic metadata collection.Native integration with AirflowStarting from Airflow version 2.7.0 OpenLineage integration is included in Airflow repository as a provider.
Theapache-airflow-providers-openlineagepackagesignificantly ease lineage tracking in Airflow,
ensuring stability by embedding the functionality directly into each provider and
simplifying the process for users to select and manage lineage collection consumers.As a result,starting from Airflow 2.7.0one should use the native Airflow Openlineage providerpackage.The ongoing development and enhancements will be focused on theapache-airflow-providers-openlineagepackage,
while theopenlineage-airflowwill primarily be updated for bug fixes.FeaturesMetadataTask lifecycleTask parametersTask runs linked toversionedcodeTask inputs / outputsLineageTrack inter-DAG dependenciesBuilt-inSQL parserLink to code builder (ex:GitHub)Metadata extractorsRequirementsPython 3.8Airflow >= 2.1,<2.8Installation$pip3installopenlineage-airflowNote:You can also addopenlineage-airflowto yourrequirements.txtfor Airflow.To install from source, run:$python3setup.pyinstallSetupAirflow 2.7+This packageshould notbe used starting with Airflow 2.7.0 andmust notbe used with Airflow 2.8+.
It was designed as Airflow's external integration that works mainly for Airflow versions <2.7.
For Airflow 2.7+ use the native Airflow OpenLineage providerpackageapache-airflow-providers-openlineage.Airflow 2.3 - 2.6The integration automatically registers itself starting from Airflow 2.3 if it's installed on the Airflow worker's Python.
This means you don't have to do anything besides configuring it, which is described in the Configuration section.Airflow 2.1 - 2.2This method has limited support: it does not support tracking failed jobs, and job starts are registered only when a job ends.Set your LineageBackend in yourairflow.cfgor via environmental variableAIRFLOW__LINEAGE__BACKENDtoopenlineage.lineage_backend.OpenLineageBackendIn contrast to integration via subclassing aDAG, aLineageBackend-based approach collects all metadata
for a task on each task's completion.The OpenLineageBackend does not take into account manually configured inlets and outlets.When enabled, the library will:On DAGstart, collect metadata for each task using anExtractorif it exists for a given operator.Collect task input / output metadata (source,schema, etc.)Collect task run-level metadata (execution time, state, parameters, etc.)On DAGcomplete, also mark the task ascompletein OpenLineageConfigurationHTTPBackend Environment Variablesopenlineage-airflowuses the OpenLineage client to push data to OpenLineage backend.The OpenLineage client depends on environment variables:OPENLINEAGE_URL- point to the service that will consume OpenLineage events.OPENLINEAGE_API_KEY- set if the consumer of OpenLineage events requires aBearerauthentication key.OPENLINEAGE_NAMESPACE- set if you are using something other than thedefaultnamespace for the job namespace.OPENLINEAGE_AIRFLOW_DISABLE_SOURCE_CODE- set toFalseif you want the source code of callables provided in the PythonOperator to be sent in OpenLineage events.For backwards compatibility,openlineage-airflowalso supports configuration viaMARQUEZ_URL,MARQUEZ_NAMESPACEandMARQUEZ_API_KEYvariables.MARQUEZ_URL=http://my_hosted_marquez.example.com:5000
MARQUEZ_NAMESPACE=my_special_nsExtractors : Sending the correct data from your DAGsIf you do nothing, the OpenLineage backend will receive theJoband theRunfrom your DAGs, but,
unless you use one of the few operators for which this integration provides an extractor, input and output metadata will not be sent.openlineage-airflowallows you to do more than that by building "Extractors." An extractor is an object
suited to extract metadata from a particular operator (or operators).Name : The name of the taskInputs : A list of input datasetsOutputs : A list of output datasetsContext : The Airflow context for the taskBundled Extractorsopenlineage-airflowprovides extractors for:PostgresOperatorMySqlOperatorAthenaOperatorBigQueryOperatorSnowflakeOperatorTrinoOperatorGreatExpectationsOperatorSFTPOperatorFTPFileTransmitOperatorPythonOperatorRedshiftDataOperator,RedshiftSQLOperatorSageMakerProcessingOperator,SageMakerProcessingOperatorAsyncSageMakerTrainingOperator,SageMakerTrainingOperatorAsyncSageMakerTransformOperator,SageMakerTransformOperatorAsyncS3CopyObjectExtractor,S3FileTransformExtractorGCSToGCSOperatorDbtCloudRunJobOperatorSQL Operators utilize the SQL parser. There is an experimental SQL parser activated if you installopenlineage-sqlon your Airflow worker.Custom ExtractorsIf your DAGs contain additional operators from which you want to extract lineage data, fear not - you can always
provide custom extractors. They should derive fromBaseExtractor.There are two ways to register them for use inopenlineage-airflow.One way is to add them to theOPENLINEAGE_EXTRACTORSenvironment variable, separated by a semi-colon(;).OPENLINEAGE_EXTRACTORS=full.path.to.ExtractorClass;full.path.to.AnotherExtractorClassTo ensure OpenLineage logging propagation to custom extractors you should useself.loginstead of creating a logger yourself.Default ExtractorWhen you own operators' code this is not necessary to provide custom extractors. You can also use Default Extractor's capability.In order to do that you should define at least one of two methods in operator:get_openlineage_facets_on_start()Extracts metadata on start of task.get_openlineage_facets_on_complete(task_instance: TaskInstance)Extracts metadata on complete of task. This should accepttask_instanceargument, similar toextract_on_completemethod in base extractors.If you don't defineget_openlineage_facets_on_completemethod it would fall back toget_openlineage_facets_on_start.Great ExpectationsThe Great Expectations integration works by providing an OpenLineageValidationAction. You need to include it into youraction_listingreat_expectations.yml.The following example illustrates a way to change the default configuration:validation_operators:action_list_operator:# To learn how to configure sending Slack notifications during evaluation# (and other customizations), read: https://docs.greatexpectations.io/en/latest/autoapi/great_expectations/validation_operators/index.html#great_expectations.validation_operators.ActionListValidationOperatorclass_name: ActionListValidationOperatoraction_list:- name: store_validation_resultaction:class_name: StoreValidationResultAction- name: store_evaluation_paramsaction:class_name: StoreEvaluationParametersAction- name: update_data_docsaction:class_name: UpdateDataDocsAction+     - name: openlineage+       action:+         class_name: OpenLineageValidationAction+         module_name: openlineage.common.provider.great_expectations.action# - name: send_slack_notification_on_validation_result#   action:#     class_name: SlackNotificationAction#     # put the actual webhook URL in the uncommitted/config_variables.yml file#     slack_webhook: ${validation_notification_slack_webhook}#     notify_on: all # possible values: "all", "failure", "success"#     renderer:#       module_name: great_expectations.render.renderer.slack_renderer#       class_name: SlackRendererIf you're usingGreatExpectationsOperator, you need to setvalidation_operator_nameto an operator that includes OpenLineageValidationAction.
Setting it ingreat_expectations.ymlfiles isn't enough - the operator overrides it with the default name if a different one is not provided.To see an example of a working configuration, seeDAGandGreat Expectations configurationin the integration tests.LoggingIn addition to conventional logging approaches, theopenlineage-airflowpackage provides an alternative way of configuring its logging behavior. By setting theOPENLINEAGE_AIRFLOW_LOGGINGenvironment variable, you can establish the logging level for theopenlineage.airflowand its child modules.Triggering Child JobsCommonly, Airflow DAGs will trigger processes on remote systems, such as an Apache Spark or Apache
Beam job. Those systems may have their own OpenLineage integrations and report their own
job runs and dataset inputs/outputs. To propagate the job hierarchy, tasks must send their own run
ids so that the downstream process can report theParentRunFacetwith the proper run id.Thelineage_run_idandlineage_parent_idmacros exists to inject the run id or whole parent run information
of a given task into the arguments sent to a remote processing job's Airflow operator. The macro requires the
DAGrun_idand the task to access the generatedrun_idfor that task. For example, a Spark job can be triggered
using theDataProcPySparkOperatorwith the correct parentrun_idusing the following configuration:t1=DataProcPySparkOperator(task_id=job_name,#required pyspark configuration,job_name=job_name,dataproc_pyspark_properties={'spark.driver.extraJavaOptions':f"-javaagent:{jar}={os.environ.get('OPENLINEAGE_URL')}/api/v1/namespaces/{os.getenv('OPENLINEAGE_NAMESPACE','default')}/jobs/{job_name}/runs/{{{{macros.OpenLineagePlugin.lineage_run_id(task, task_instance)}}}}?api_key={os.environ.get('OPENLINEAGE_API_KEY')}"dag=dag)Secrets redactionThe integration uses Airflow SecretsMasker to hide secrets from produced metadata events. As not all fields in the metadata should be redacted,RedactMixinis used to pass information about which fields should be ignored by the process.Typically, you should subclassRedactMixinand use the_skip_redactattribute as a list of names of fields to be skipped.However, all facets inheriting fromBaseFacetshould use the_additional_skip_redactattribute as an addition to the regular list (['_producer', '_schemaURL']).DevelopmentTo install all dependencies forlocaldevelopment:The Airflow integration depends onopenlineage.sql,openlineage.commonandopenlineage.client.python. You should install them first independently or try to install them with following command:$pipinstall-rdev-requirements.txtThere is also a bash script that can run an arbitrary Airflow image with an OpenLineage integration built from the current branch. Additionally, it mounts OpenLineage Python packages as Docker volumes. This enables you to change your code without the need to constantly rebuild Docker images to run tests.
Run it as:$AIRFLOW_IMAGE=<airflow_image_with_tag>./scripts/run-dev-airflow.sh[--help]Unit testsTo run the entire unit test suite, use the below command:$toxor choose one of the environments, e.g.:$tox-epy-airflow214You can also skip usingtoxand runpyteston your own dev environment.Integration testsThe integration tests require the use ofdocker compose. There are scripts prepared to make build images and run tests easier.$AIRFLOW_IMAGE=<name-of-airflow-image>./tests/integration/docker/up.sh$AIRFLOW_IMAGE=apache/airflow:2.3.1-python3.8./tests/integration/docker/up.shWhen usingrun-dev-airflow.sh, you can add the-iflag or--attach-integrationflag to run integration tests in a dev environment.
This can be helpful when you need to run arbitrary integration tests during development. For example, the following command run in the integration container...python-mpytesttest_integration.py::test_integration[great_expectations_validation-requests/great_expectations.json]...runs a single test which you can repeat after changes in code.SPDX-License-Identifier: Apache-2.0Copyright 2018-2023 contributors to the OpenLineage project