Package: apache-airflow-providers-isolation
Isolation ProviderRuntime Operator Isolation in Airflow.Created with ❤️ by the CSE Team @ AstronomerSummaryQuickstartPre-requisites:StepsInstallationCLIUsageHowWhat it isWhat it isn'tRequirementsCLIHost AirflowTarget Isolated EnvironmentSummaryThe Isolation Provider provides theIsolatedOperatorand theisolationctlCLI.
It provides the capacity to run any Airflow Operator in an isolated fashion.Why useIsolatedOperator?Run a different version of an underlying library,
between separate teams on the same Airflow instance,
even between separate tasks in the same DAGRun entirely separate versions of Python for a taskKeep "heavy" dependencies separate from AirflowRun an Airflow task in a completely separate environment - or on a server all the way across the world.Run a task "safely" - separate from the Airflow instanceRun a task with dependencies that conflict more easilyDo all of the above while having unmodified access to (almost) all of 'normal' Airflow -
operators, XCOMs, logs, deferring, callbacks.What does theisolationctlprovide?theisolationctlgives you an easy way to manage "environments" for yourIsolatedOperatorWhen shouldn't you useIsolatedOperator?If youcanuse un-isolated Airflow Operators, you stillshoulduse un-isolated Airflow Operators.'Talking back' to Airflow is no longer possible in anIsolatedOperator.
You cannot doVariable.setwithin anIsolatedOperator(operator=PythonOperator), nor can you query the Airflow
Metadata Database.QuickstartThis quickstart utilizes a local Kubernetes cluster, and a local image registry to host isolated environments.Pre-requisites:Dockeris installedA local Kubernetes (
e.g.Docker Desktop,Minikube,
etc) is runningastroCLIis installedStepsDownload and install theisolationctlCLI viapipinstallapache-airflow-providers-isolation[cli]Set up the project:isolationctlinit--example--local--local-registry--astro--git--dependency--exampleaddsenvironments/example--localuseskube config viewto add aKUBERNETES_DEFAULTAirflow Connection in.env--local-registryruns a docker image registry atlocalhost:5000--astrorunsastro dev init--gitrunsgit init--dependencyaddsapache-airflow-providers-isolation[kubernetes]torequirements.txtAdd an 'older' version of Pandas toenvironments/example/requirements.txtecho"\npandas==1.4.2">>environments/example/requirements.txtecho"\nbuild-essential">>environments/example/packages.txtAdd the http provider toenvironments/example/requirements.txtpushdenvironments/example/
astroregistryprovideraddhttppopdBuild theexampleenvironment and deploy it to the local registryisolationctldeploy--local-registryAdd the Kubernetes Provider to the Astro project (not required- it is a transitive dependency - but always good to
be explicit)astroregistryprovideraddkubernetesDisable OpenLineage just to clean up the logging, locally, for the example. Then start the Airflow Project.echo"\nOPENLINEAGE_DISABLED=true">>.env
astrodevstart--no-browserRun the example DAGastrodevrundagstestisolation_provider_example_dag🎉🎉🎉Installationpipinstallapache-airflow-providers-isolation[kubernetes]CLITo install theisolationctl CLIpipinstallapache-airflow-providers-isolation[cli]UsagefromdatetimeimportdatetimefromairflowimportDAGfromairflow.operators.bashimportBashOperatorfromisolation.operators.isolationimportIsolatedOperatorwithDAG("isolation_provider_example_dag",schedule=None,start_date=datetime(1970,1,1),):IsolatedOperator(task_id="echo_with_bash_operator",operator=BashOperator,environment="example",bash_command="echo {{ params.hi }}",)HowWhat it isTheIsolatedOperatoris a wrapper on top of underlying operator,
such asKubernetesPodOperatoror potentially other operators,
to run someOtherOperatorin an isolated environment without any need to know or understand how the operator
underlying theIsolatedOperatorworks - beyond what is absolutely required to run.It also has theisolationctl CLIwhich aids in the creation and deployment of isolated environments to run
in.The steps that theIsolatedOperatortakes, in more detail:Airflow initializes an underlying base operator (such asKubernetesPodOperator)
based on theIsolatedOperator.__init__Some amount of initial state (e.g. the Airflow, Connections, Variables)
will be provided to the Isolated environment via special ENV varsThe underlying operator (such asKubernetesPodOperator) executes as normalThe isolated environment runs within the underlying operator,
and is bootstrapped via thePostIsolationHook.State is re-establish via passed in ENV varsIn an otherwise un-initialized Airflow, via an internal hok, theOtherOperatoris executedThis isolated environment's Airflowhas no knowledgeof the parent Airflow that launched it,
it should have no access to the parents' Metadata Database,
andit should not be able to communicate back to the parent Airflow.Most Airflow functionality should work out-of-the-box, simply due to the reliance on the underlying operators to do most
of the "heavy lifting" - e.g. XCOMs and LogsWhat it isn'tIf youcanuse un-isolated Airflow Operators, you stillshoulduse un-isolated Airflow Operators.
This won't make an operator thatwould run normallyany easier to run.Anything that requirescommunicating backto the parent Airflow at runtime is unsupported or impossible -
e.g.Variable.set(...)Anything that requires querying or modifying the parent Airflow's state at runtime is unsupported or impossible.
Examples include@provide_session, orTaskInstance.query(...), orExternalTaskSensor, orTriggerDagRunOperatorIt's possible other less-traditional parts of Airflow may not yet be supported,
due to development effort - e.g.@taskannotations or Airflow 2.7 Setup and Teardowns - depending on how precisely they are invoked.It is possible that things likeon_failure_callbacks or lineage data may not work - depending on how exactly they
are invoked -
but if these things work with via the underlying operator and are set on the underlying operator,
then they should work with this.RequirementsCLIPython ">=3.8,<3.12"You must have write access to a Container Registry, read more atCLI RequirementsHost AirflowPython ">=3.8,<3.12"Airflow >=2.3Must have access to create containers in the target Kubernetes EnvironmentTarget Isolated EnvironmentPython ">=3.8,<3.12"Airflow >=2.3Note: Airflow 2.2 doesn't havetest_modefor executing tasks - which is currently utilized to bypass setup of
the
isolated environment's AirflowMust be a Kubernetes environment