Package: airflow-google-cloud-run-plugin
airflow-google-cloud-run-pluginAirflow plugin for
orchestratingGoogle Cloud Run jobs.FeaturesEasier to use alternative
toKubernetesPodOperatorSecurely use sensitive data stored in Google Cloud Secrets ManagerCreate tasks with isolated dependenciesEnables polyglot workflowsResourcesCore OperatorsCloudRunJobOperatorCRUD-Based OperatorsCloudRunCreateJobOperatorCloudRunGetJobOperator🔜CloudRunUpdateJobOperator🔜CloudRunDeleteJobOperatorCloudRunListJobsOperator🔜HooksCloudRunJobHookSensorsCloudRunJobExecutionSensor🔜UsageSimple Job LifecyclefromairflowimportDAGfromairflow_google_cloud_run_plugin.operators.cloud_runimportCloudRunJobOperatorwithDAG(dag_id="example_dag")asdag:job=CloudRunJobOperator(task_id="example-job",name="example-job",location="us-central1",project_id="example-project",image="gcr.io/gcp-runtimes/ubuntu_18_0_4",command=["echo"],cpu="1000m",memory="512Mi",create_if_not_exists=True,delete_on_exit=True)CRUD Job LifecyclefromairflowimportDAGfromairflow_google_cloud_run_plugin.operators.cloud_runimport(CloudRunJobOperator,CloudRunCreateJobOperator,CloudRunDeleteJobOperator,)withDAG(dag_id="example_dag")asdag:create_job=CloudRunCreateJobOperator(task_id="create",name="example-job",location="us-central1",project_id="example-project",image="gcr.io/gcp-runtimes/ubuntu_18_0_4",command=["echo"],cpu="1000m",memory="512Mi")run_job=CloudRunJobOperator(task_id="run",name="example-job",location="us-central1",project_id="example-project")delete_job=CloudRunDeleteJobOperator(task_id="delete",name="example-job",location="us-central1",project_id="example-project")create_job>>run_job>>delete_jobUsing Environment VariablesfromairflowimportDAGfromairflow_google_cloud_run_plugin.operators.cloud_runimportCloudRunJobOperator# Simple environment variableFOO={"name":"FOO","value":"not_so_secret_value_123"}# Environment variable from Secret ManagerBAR={"name":"BAR","valueFrom":{"secretKeyRef":{"name":"super_secret_password","key":"1"# or "latest" for latest secret version}}}withDAG(dag_id="example_dag")asdag:job=CloudRunJobOperator(task_id="example-job",name="example-job",location="us-central1",project_id="example-project",image="gcr.io/gcp-runtimes/ubuntu_18_0_4",command=["echo"],args=["$FOO","$BAR"],env_vars=[FOO,BAR],cpu="1000m",memory="512Mi",create_if_not_exists=True,delete_on_exit=True)Improvement SuggestionsAdd support for Cloud Run servicesNicer user experience for defining args and commandsUse approach from other GCP operators once this issue is resolvedhttps://github.com/googleapis/python-run/issues/64Add operators for all CRUD operationsAdd run sensor (seelink)Enable volume mounts (seeTaskSpec)Allow user to configure resource requirementsrequests(
seeResourceRequirements)Add remaining container options (seeContainer)Allow non-default credentials and for user to specify service account (
seelink)Allow failure threshold. If more than one task is specified, user should be allowed to specify number of failures
allowedAdd custom links for log URIsAdd wrapper class for easier environment variable definition. Similar toSecretfrom Kubernetes provider (
seelink)Add slight time padding between job create and runAdd ability to choose to replace the job with new config values if values have changed