Package: openlineage-dagster
NoteNew integration maintainers are needed! Pleaseopen an issueto get started.OpenLineage Dagster IntegrationA library that integratesDagsterwithOpenLineagefor automatic metadata collection.
It provides an OpenLineage sensor, a Dagster sensor that tails Dagster event logs for tracking metadata.
On each sensor evaluation, the function processes a batch of event logs, converts Dagster events into OpenLineage events,
and emits them to an OpenLineage backend.FeaturesMetadataDagster job & op lifecycleRequirementsPython 3.8Dagster 1.0.0+Installation$python-mpipinstallopenlineage-dagsterUsageOpenLineage Sensor & Event Log Storage RequirementsSingle OpenLineage sensor per Dagster instanceAs the OpenLineage sensor processes all event logs for a given Dagster instance, define and enable only one sensor per instance.
Running multiple sensors will result in duplicate OpenLineage job runs being emitted for Dagster steps with different OpenLineage run IDs. These are dynamically generated during sensor runs.Non-shardedEvent Log StorageFor the sensor to handle all event logs across runs, use non-sharded event log storage.
If an event log storage sharded by run (i.e., the defaultSqliteEventLogStorage) is used, the cursor that tracks the last processed event storage ID may not update properly.OpenLineage Sensor SetupGet a OpenLineage sensor definition from theopenlineage_sensor()factory function and add it to your Dagster repository.fromdagsterimportrepositoryfromopenlineage.dagster.sensorimportopenlineage_sensor@repositorydefmy_repository():openlineage_sensor_def=openlineage_sensor()returnother_defs+[openlineage_sensor_def]Given that parallel sensor runs are not supported at the time of writing, some tuning may be necessary to avoid affecting other sensors' performance.See Dagster's documentation onEvaluation Intervalfor more detail onminimum_interval_seconds, which defaults to 30 seconds.record_filter_limitis the maximum number of event logs to process on each sensor evaluation, and it defaults to 30 records per evaluation.
Default values can be overridden:@repositorydefmy_repository():openlineage_sensor_def=openlineage_sensor(minimum_interval_seconds=60,record_filter_limit=60,)returnother_defs+[openlineage_sensor_def]The OpenLineage sensor handles event logs in ascending order of storage ID and starts with the first log by default.
Optionally,after_storage_idcan be specified to customize the starting point.
This is only applicable when the cursor is undefined or has been deleted.@repositorydefmy_repository():openlineage_sensor_def=openlineage_sensor(after_storage_id=100)returnother_defs+[openlineage_sensor_def]OpenLineage Adapter & Client ConfigurationThe sensor uses an OpenLineage adapter and client to convert and push data to an OpenLineage backend.
These depend on environment variables.If using User Repository Deployments, add the below variables to the repository where the sensor is defined.
Otherwise, add the variables to the Dagster Daemon.OPENLINEAGE_URL- point to the service which will consume OpenLineage events.OPENLINEAGE_API_KEY- set if the consumer of OpenLineage events requires aBearerauthentication key.OPENLINEAGE_NAMESPACE- set if you are using something other than thedefaultas the default namespace when a Dagster repository is undefined.OpenLineage Namespace & Dagster RepositoryFor Dagster jobs organized in repositories, Dagster keeps track of the repository name for each pipeline run.
When the repository name is present, it is always used as the OpenLineage namespace name.OPENLINEAGE_NAMESPACEoption is a way to fall back and provide some other static default value.DevelopmentTo install all dependencies for local development:$python-mpipinstall-e.[dev]# or python -m pip install -e .\[dev\] in zshTo run the test suite:$pytestSPDX-License-Identifier: Apache-2.0Copyright 2018-2023 contributors to the OpenLineage project