Repository: apache/airflow

Issue:
Invalid livenessProbe for Standalone DAG Processor
### Official Helm Chart version

1.7.0 (latest released)

### Apache Airflow version

2.3.4

### Kubernetes Version

1.22.12-gke.1200	

### Helm Chart configuration

```yaml
  dagProcessor:
    enabled: true
```

### Docker Image customisations

```dockerfile
FROM apache/airflow:2.3.4-python3.9

USER root
RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
RUN curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
RUN apt-get update && apt-get install -y google-cloud-cli
RUN curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
RUN sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
USER airflow
```

### What happened

Current DAG Processor livenessProbe is the following:
```
CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
    airflow jobs check --hostname $(hostname)
```
This command checks the metadata DB searching for an active job whose hostname is the current pod's one (_airflow-dag-processor-xxxx_). 
However, after running the dag-processor pod for more than 1 hour, there are no jobs with the processor hostname in the jobs table.
![image](https://user-images.githubusercontent.com/28935464/196711859-98dadb8f-3273-42ec-a4db-958890db34b7.png)
![image](https://user-images.githubusercontent.com/28935464/196711947-5a0fc5d7-4b91-4e82-9ff0-c721e6a4c1cd.png)

As a consequence, the livenessProbe fails and the pod is constantly restarting.

After investigating the code, I found out that DagFileProcessorManager is not creating jobs in the metadata DB, so the livenessProbe is not valid.

### What you think should happen instead

A new job should be created for the Standalone DAG Processor.
By doing that, the _airflow jobs check --hostname <hostname>_ command would work correctly and the livenessProbe wouldn't fail

### How to reproduce

1. Deploy airflow with a standalone dag-processor.
2. Wait for ~ 5 minutes
3. Check that the livenessProbe has been failing for 5 minutes and the pod has been restarted.

### Anything else

I think this behavior is inherited from the NOT standalone dag-processor mode (the livenessProbe checks for a SchedulerJob, that in fact contains the "DagProcessorJob")

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
A temporary workaround for this is to set liveness probe command to `exit 0`.

in terraform `helm_release`:
```
...
  set {
    name  = "dagProcessor.livenessProbe.command[0]"
    value = "sh"
  }
  set {
    name  = "dagProcessor.livenessProbe.command[1]"
    value = "-c"
  }
  set {
    name  = "dagProcessor.livenessProbe.command[2]"
    value = "exit 0"
  }
```
or in helm deploy:
```
--set dagProcessor.livenessProbe.command[0] ...
```
and btw, documentation is missing the liveness probe command entry. Found it is overridable in the chart source code :)
Yeah @george-zubrienko, however, disabling a livenessProbe is something dangerous: the pod wouldn't be restarted if the processor freezes.
In our case we would see this in Datadog alerts. It is better than a non-working liveness probe that in the end goes in crashloopback and you get delays in dag processing.
our fix for the moment is to use the non-standalone dag-processor ;)
Yeah but this feature is long due to be fair. No problem re-enabling the probe when it works - crossing fingers this issue gets attention :)
Assigned it for 1.8.0.
@george-zubrienko  - feel free to make PR for that - then you will not have to wait and become one of the > 2200 contributors. It's easy!
@potiuk thanks for the offer and I'd love to when I have a bit more free time for OS work. I won't feel great if I commit to doing a PR and then go afk for couple weeks because there are issues in other projects (also OS ones) that need to be addressed. So not for this issue, but for future ones, I'll consider for sure :)
I have tested that removing --hostname resolves the issue here #28600.

I wonder if my approach is good then I will add unittest.

Please review and let me know if there are any further changes needed.
Hi @farhan0syakir 
I think removing _hostname_ works because the command just looks if there are jobs in the Metadata DB, no matter which kind of job they are (schedulerJob, triggererJob, etc.) .
However, that makes the livenessProbe useless, as it's not checking that the dagProcessor is working.
Thanks for pointing that out @csp33 
I closed the pull request, I might try another approach later

Resolving patch:
diff --git a/airflow/cli/commands/dag_processor_command.py b/airflow/cli/commands/dag_processor_command.py
--- a/airflow/cli/commands/dag_processor_command.py
+++ b/airflow/cli/commands/dag_processor_command.py
@@ -25,21 +25,21 @@
 
 from airflow import settings
 from airflow.configuration import conf
-from airflow.dag_processing.manager import DagFileProcessorManager
+from airflow.jobs.dag_processor_job import DagProcessorJob
 from airflow.utils import cli as cli_utils
 from airflow.utils.cli import setup_locations, setup_logging
 
 log = logging.getLogger(__name__)
 
 
-def _create_dag_processor_manager(args) -> DagFileProcessorManager:
+def _create_dag_processor_job(args) -> DagProcessorJob:
     """Creates DagFileProcessorProcess instance."""
     processor_timeout_seconds: int = conf.getint("core", "dag_file_processor_timeout")
     processor_timeout = timedelta(seconds=processor_timeout_seconds)
-    return DagFileProcessorManager(
+    return DagProcessorJob(
+        processor_timeout=processor_timeout,
         dag_directory=args.subdir,
         max_runs=args.num_runs,
-        processor_timeout=processor_timeout,
         dag_ids=[],
         pickle_dags=args.do_pickle,
     )
@@ -55,7 +55,7 @@ def dag_processor(args):
     if sql_conn.startswith("sqlite"):
         raise SystemExit("Standalone DagProcessor is not supported when using sqlite.")
 
-    manager = _create_dag_processor_manager(args)
+    job = _create_dag_processor_job(args)
 
     if args.daemon:
         pid, stdout, stderr, log_file = setup_locations(
@@ -74,10 +74,6 @@ def dag_processor(args):
                 umask=int(settings.DAEMON_UMASK, 8),
             )
             with ctx:
-                try:
-                    manager.start()
-                finally:
-                    manager.terminate()
-                    manager.end()
+                job.run()
     else:
-        manager.start()
+        job.run()
diff --git a/airflow/jobs/dag_processor_job.py b/airflow/jobs/dag_processor_job.py
new file mode 100644
--- /dev/null
+++ b/airflow/jobs/dag_processor_job.py
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import os
+from datetime import timedelta
+
+from airflow.dag_processing.manager import DagFileProcessorManager
+from airflow.jobs.base_job import BaseJob
+
+
+class DagProcessorJob(BaseJob):
+    """
+    :param dag_directory: Directory where DAG definitions are kept. All
+        files in file_paths should be under this directory
+    :param max_runs: The number of times to parse and schedule each file. -1
+        for unlimited.
+    :param processor_timeout: How long to wait before timing out a DAG file processor
+    :param dag_ids: if specified, only schedule tasks with these DAG IDs
+    :param pickle_dags: whether to pickle DAGs.
+    :param async_mode: Whether to start agent in async mode
+    """
+
+    __mapper_args__ = {"polymorphic_identity": "DagProcessorJob"}
+
+    def __init__(
+        self,
+        dag_directory: os.PathLike,
+        max_runs: int,
+        processor_timeout: timedelta,
+        dag_ids: list[str] | None,
+        pickle_dags: bool,
+        *args,
+        **kwargs,
+    ):
+        self.processor = DagFileProcessorManager(
+            dag_directory=dag_directory,
+            max_runs=max_runs,
+            processor_timeout=processor_timeout,
+            dag_ids=dag_ids,
+            pickle_dags=pickle_dags,
+        )
+        super().__init__(*args, **kwargs)
+
+    def _execute(self) -> None:
+        self.log.info("Starting the Dag Processor Job")
+        try:
+            self.processor.start()
+        except Exception:
+            self.log.exception("Exception when executing DagProcessorJob")
+            raise
+        finally:
+            self.processor.terminate()
+            self.processor.end()


Test patch:
diff --git a/tests/cli/commands/test_dag_processor_command.py b/tests/cli/commands/test_dag_processor_command.py
--- a/tests/cli/commands/test_dag_processor_command.py
+++ b/tests/cli/commands/test_dag_processor_command.py
@@ -42,16 +42,17 @@ def setup_class(cls):
             ("core", "load_examples"): "False",
         }
     )
-    @mock.patch("airflow.cli.commands.dag_processor_command.DagFileProcessorManager")
+    @mock.patch("airflow.cli.commands.dag_processor_command.DagProcessorJob")
     @pytest.mark.skipif(
         conf.get_mandatory_value("database", "sql_alchemy_conn").lower().startswith("sqlite"),
         reason="Standalone Dag Processor doesn't support sqlite.",
     )
-    def test_start_manager(
+    def test_start_job(
         self,
-        mock_dag_manager,
+        mock_dag_job,
     ):
         """Ensure that DagFileProcessorManager is started"""
-        args = self.parser.parse_args(["dag-processor"])
-        dag_processor_command.dag_processor(args)
-        mock_dag_manager.return_value.start.assert_called()
+        with conf_vars({("scheduler", "standalone_dag_processor"): "True"}):
+            args = self.parser.parse_args(["dag-processor"])
+            dag_processor_command.dag_processor(args)
+            mock_dag_job.return_value.run.assert_called()