Repository: apache/airflow
Pull request #49031: Bring back serve_logs to be in the core
Author: login: potiuk
Labels: name: area:CLI
name: area:providers
name: area:Triggerer
name: provider:celery
Created: 2025-04-10T04:16:16Z | Merged: 2025-04-10T09:05:54Z

The `serve_logs` has been moved in #48457 to fab provider by mistake. It should remain in the airflow-core. We still depend on flask in the core (but as a next step we can replace it by starlette and unicorn).

Fixes: #49028

<!--
 Licensed to the Apache Software Foundation (ASF) under one
 or more contributor license agreements.  See the NOTICE file
 distributed with this work for additional information
 regarding copyright ownership.  The ASF licenses this file
 to you under the Apache License, Version 2.0 (the
 "License"); you may not use this file except in compliance
 with the License.  You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing,
 software distributed under the License is distributed on an
 "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 KIND, either express or implied.  See the License for the
 specific language governing permissions and limitations
 under the License.
 -->

<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.

Feel free to ping committers for the review!

In case of an existing issue, reference it using one of the following:

closes: #ISSUE
related: #ISSUE

How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->



<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).



Diff:
diff --git a/airflow-core/pyproject.toml b/airflow-core/pyproject.toml
index 527ddec8d854a..3d77dee53f999 100644
--- a/airflow-core/pyproject.toml
+++ b/airflow-core/pyproject.toml
@@ -84,7 +84,10 @@ dependencies = [
     # 0.115.10 fastapi was a bad release that broke our API's and static checks.
     # Related fastapi issue here: https://github.com/fastapi/fastapi/discussions/13431
     "fastapi[standard]>=0.115.0,!=0.115.10",
+    # We could get rid of flask and gunicorn if we replace serve_logs with a starlette + unicorn
+    "flask>=2.1.1",
     "gitpython>=3.1.40",
+    # We could get rid of flask and gunicorn if we replace serve_logs with a starlette + unicorn
     "gunicorn>=20.1.0",
     "httpx>=0.25.0",
     'importlib_metadata>=6.5;python_version<"3.12"',
diff --git a/airflow-core/src/airflow/cli/commands/scheduler_command.py b/airflow-core/src/airflow/cli/commands/scheduler_command.py
index 37cb4847f3824..15093fd24d0c7 100644
--- a/airflow-core/src/airflow/cli/commands/scheduler_command.py
+++ b/airflow-core/src/airflow/cli/commands/scheduler_command.py
@@ -29,7 +29,6 @@
 from airflow.executors.executor_loader import ExecutorLoader
 from airflow.jobs.job import Job, run_job
 from airflow.jobs.scheduler_job_runner import SchedulerJobRunner
-from airflow.providers.celery.version_compat import AIRFLOW_V_3_0_PLUS
 from airflow.utils import cli as cli_utils
 from airflow.utils.providers_configuration_loader import providers_configuration_loaded
 from airflow.utils.scheduler_health import serve_health_check
@@ -60,18 +59,7 @@ def scheduler(args: Namespace):
 
 @contextmanager
 def _serve_logs(skip_serve_logs: bool = False):
-    """Start serve_logs sub-process."""
-    if AIRFLOW_V_3_0_PLUS:
-        try:
-            from airflow.providers.fab.www.serve_logs import serve_logs
-        except ImportError:
-            raise ImportError(
-                "Celery requires FAB provider to be installed in order to run this command. "
-                "Please install the FAB provider by running: "
-                "pip install apache-airflow-providers-celery[fab]"
-            )
-    else:
-        from airflow.utils.serve_logs import serve_logs  # type: ignore[no-redef]
+    from airflow.utils.serve_logs import serve_logs
 
     sub_proc = None
     executor_class, _ = ExecutorLoader.import_default_executor_cls()
diff --git a/airflow-core/src/airflow/cli/commands/triggerer_command.py b/airflow-core/src/airflow/cli/commands/triggerer_command.py
index b953006e5c996..c5f1682c6de54 100644
--- a/airflow-core/src/airflow/cli/commands/triggerer_command.py
+++ b/airflow-core/src/airflow/cli/commands/triggerer_command.py
@@ -28,7 +28,6 @@
 from airflow.configuration import conf
 from airflow.jobs.job import Job, run_job
 from airflow.jobs.triggerer_job_runner import TriggererJobRunner
-from airflow.providers.celery.version_compat import AIRFLOW_V_3_0_PLUS
 from airflow.utils import cli as cli_utils
 from airflow.utils.providers_configuration_loader import providers_configuration_loaded
 
@@ -36,17 +35,7 @@
 @contextmanager
 def _serve_logs(skip_serve_logs: bool = False) -> Generator[None, None, None]:
     """Start serve_logs sub-process."""
-    if AIRFLOW_V_3_0_PLUS:
-        try:
-            from airflow.providers.fab.www.serve_logs import serve_logs
-        except ImportError:
-            raise ImportError(
-                "Celery requires FAB provider to be installed in order to run this command. "
-                "Please install the FAB provider by running: "
-                "pip install apache-airflow-providers-celery[fab]"
-            )
-    else:
-        from airflow.utils.serve_logs import serve_logs  # type: ignore[no-redef]
+    from airflow.utils.serve_logs import serve_logs
 
     sub_proc = None
     if skip_serve_logs is False:
diff --git a/providers/fab/src/airflow/providers/fab/www/serve_logs.py b/airflow-core/src/airflow/utils/serve_logs.py
similarity index 96%
rename from providers/fab/src/airflow/providers/fab/www/serve_logs.py
rename to airflow-core/src/airflow/utils/serve_logs.py
index 9a6919fc0723c..32f71ffd99ab2 100644
--- a/providers/fab/src/airflow/providers/fab/www/serve_logs.py
+++ b/airflow-core/src/airflow/utils/serve_logs.py
@@ -21,6 +21,7 @@
 import logging
 import os
 import socket
+import sys
 from collections import namedtuple
 
 import gunicorn.app.base
@@ -32,7 +33,6 @@
     InvalidIssuedAtError,
     InvalidSignatureError,
 )
-from setproctitle import setproctitle
 from werkzeug.exceptions import HTTPException
 
 from airflow.api_fastapi.auth.tokens import JWTValidator, get_signing_key
@@ -169,6 +169,13 @@ def load(self):
 
 def serve_logs(port=None):
     """Serve logs generated by Worker."""
+    # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+    os_type = sys.platform
+    if os_type == "darwin":
+        logger.debug("Mac OS detected, skipping setproctitle")
+    else:
+        from setproctitle import setproctitle
+
     setproctitle("airflow serve-logs")
     wsgi_app = create_app()
 
diff --git a/airflow-core/tests/unit/cli/commands/test_scheduler_command.py b/airflow-core/tests/unit/cli/commands/test_scheduler_command.py
index f10faa87e3b75..8c6958159d0b0 100644
--- a/airflow-core/tests/unit/cli/commands/test_scheduler_command.py
+++ b/airflow-core/tests/unit/cli/commands/test_scheduler_command.py
@@ -25,8 +25,8 @@
 from airflow.cli import cli_parser
 from airflow.cli.commands import scheduler_command
 from airflow.executors import executor_loader
-from airflow.providers.fab.www.serve_logs import serve_logs
 from airflow.utils.scheduler_health import serve_health_check
+from airflow.utils.serve_logs import serve_logs
 
 from tests_common.test_utils.config import conf_vars
 
diff --git a/providers/fab/tests/unit/fab/www/test_serve_logs.py b/airflow-core/tests/unit/utils/test_serve_logs.py
similarity index 98%
rename from providers/fab/tests/unit/fab/www/test_serve_logs.py
rename to airflow-core/tests/unit/utils/test_serve_logs.py
index 6f46543515204..98d19c93b44ff 100644
--- a/providers/fab/tests/unit/fab/www/test_serve_logs.py
+++ b/airflow-core/tests/unit/utils/test_serve_logs.py
@@ -26,8 +26,8 @@
 
 from airflow.api_fastapi.auth.tokens import JWTGenerator
 from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
-from airflow.providers.fab.www.serve_logs import create_app
 from airflow.utils import timezone
+from airflow.utils.serve_logs import create_app
 
 from tests_common.test_utils.config import conf_vars
 
@@ -81,7 +81,7 @@ def sample_log(request, tmp_path):
         base_log_dir = Path(DEFAULT_LOGGING_CONFIG["handlers"]["task"]["base_log_folder"])
     else:
         raise ValueError(f"Unknown client fixture: {client}")
-
+    base_log_dir.mkdir(exist_ok=True, parents=True)
     f = base_log_dir.joinpath("sample.log")
     f.write_text(LOG_DATA)
     return f
diff --git a/dev/breeze/tests/test_selective_checks.py b/dev/breeze/tests/test_selective_checks.py
index b8ce1acba943c..9fb0d99d760f7 100644
--- a/dev/breeze/tests/test_selective_checks.py
+++ b/dev/breeze/tests/test_selective_checks.py
@@ -2157,7 +2157,7 @@ def test_upgrade_to_newer_dependencies(
         ),
         pytest.param(
             ("providers/celery/src/airflow/providers/celery/file.py",),
-            {"docs-list-as-string": "celery cncf.kubernetes fab"},
+            {"docs-list-as-string": "celery cncf.kubernetes"},
             id="Celery python files changed",
         ),
         pytest.param(
diff --git a/providers/celery/README.rst b/providers/celery/README.rst
index d2c6bfc722538..73789e85eb671 100644
--- a/providers/celery/README.rst
+++ b/providers/celery/README.rst
@@ -75,7 +75,6 @@ You can install such cross-provider dependencies when installing from PyPI. For
 Dependent package                                                                                                       Extra
 ======================================================================================================================  ===================
 `apache-airflow-providers-cncf-kubernetes <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes>`_  ``cncf.kubernetes``
-`apache-airflow-providers-fab <https://airflow.apache.org/docs/apache-airflow-providers-fab>`_                          ``fab``
 ======================================================================================================================  ===================
 
 The changelog for the provider package can be found in the
diff --git a/providers/celery/pyproject.toml b/providers/celery/pyproject.toml
index 1f002d01bc9e6..afcb58fbc6eb5 100644
--- a/providers/celery/pyproject.toml
+++ b/providers/celery/pyproject.toml
@@ -72,9 +72,6 @@ dependencies = [
 "cncf.kubernetes" = [
     "apache-airflow-providers-cncf-kubernetes>=7.4.0",
 ]
-"fab" = [
-    "apache-airflow-providers-fab>=2.0.0",
-]
 
 [dependency-groups]
 dev = [
@@ -82,7 +79,6 @@ dev = [
     "apache-airflow-task-sdk",
     "apache-airflow-devel-common",
     "apache-airflow-providers-cncf-kubernetes",
-    "apache-airflow-providers-fab",
     # Additional devel dependencies (do not remove this line and add extra development dependencies)
 ]
 
diff --git a/providers/celery/src/airflow/providers/celery/cli/celery_command.py b/providers/celery/src/airflow/providers/celery/cli/celery_command.py
index c7dfcf6c04d92..c3cdc793ace9e 100644
--- a/providers/celery/src/airflow/providers/celery/cli/celery_command.py
+++ b/providers/celery/src/airflow/providers/celery/cli/celery_command.py
@@ -107,17 +107,7 @@ def flower(args):
 @contextmanager
 def _serve_logs(skip_serve_logs: bool = False):
     """Start serve_logs sub-process."""
-    if AIRFLOW_V_3_0_PLUS:
-        try:
-            from airflow.providers.fab.www.serve_logs import serve_logs
-        except ImportError:
-            raise ImportError(
-                "Celery requires FAB provider to be installed in order to run this command. "
-                "Please install the FAB provider by running: "
-                "pip install apache-airflow-providers-celery[fab]"
-            )
-    else:
-        from airflow.utils.serve_logs import serve_logs  # type: ignore[no-redef]
+    from airflow.utils.serve_logs import serve_logs
 
     sub_proc = None
     if skip_serve_logs is False:


Reviews:
user: login: potiuk
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-10T04:17:23Z
user: login: ashb
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-10T05:19:33Z
user: login: amoghrajesh
body: LGTM +1
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T06:13:27Z
user: login: kaxil
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-10T06:29:39Z
user: login: eladkal
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T08:32:55Z
user: login: kaxil
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-10T08:35:43Z
user: login: eladkal
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T08:42:46Z

Review comments:
diff_hunk: @@ -81,7 +81,7 @@ def sample_log(request, tmp_path):
         base_log_dir = Path(DEFAULT_LOGGING_CONFIG["handlers"]["task"]["base_log_folder"])
     else:
         raise ValueError(f"Unknown client fixture: {client}")
-
+    base_log_dir.mkdir(exist_ok=True, parents=True)
path: airflow-core/tests/unit/utils/test_serve_logs.py
user: login: potiuk
body: Found out this was missing if the test was run standalone.
created_at: 2025-04-10T04:17:23Z
author_association: MEMBER
diff_hunk: @@ -169,6 +169,13 @@ def load(self):
 
 def serve_logs(port=None):
     """Serve logs generated by Worker."""
+    # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+    os_type = sys.platform
+    if os_type == "darwin":
+        logger.debug("Mac OS detected, skipping setproctitle")
+    else:
+        from setproctitle import setproctitle
+
path: airflow-core/src/airflow/utils/serve_logs.py
user: login: amoghrajesh
body: Needs rebasing i think cos this shouldve landed.
created_at: 2025-04-10T06:13:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -107,17 +107,7 @@ def flower(args):
 @contextmanager
 def _serve_logs(skip_serve_logs: bool = False):
     """Start serve_logs sub-process."""
-    if AIRFLOW_V_3_0_PLUS:
-        try:
-            from airflow.providers.fab.www.serve_logs import serve_logs
-        except ImportError:
-            raise ImportError(
-                "Celery requires FAB provider to be installed in order to run this command. "
-                "Please install the FAB provider by running: "
-                "pip install apache-airflow-providers-celery[fab]"
-            )
-    else:
-        from airflow.utils.serve_logs import serve_logs  # type: ignore[no-redef]
+    from airflow.utils.serve_logs import serve_logs
path: providers/celery/src/airflow/providers/celery/cli/celery_command.py
user: login: eladkal
body: Isn't this a breaking change for celery?
That import would not work on Airflow 2.9
created_at: 2025-04-10T08:32:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -107,17 +107,7 @@ def flower(args):
 @contextmanager
 def _serve_logs(skip_serve_logs: bool = False):
     """Start serve_logs sub-process."""
-    if AIRFLOW_V_3_0_PLUS:
-        try:
-            from airflow.providers.fab.www.serve_logs import serve_logs
-        except ImportError:
-            raise ImportError(
-                "Celery requires FAB provider to be installed in order to run this command. "
-                "Please install the FAB provider by running: "
-                "pip install apache-airflow-providers-celery[fab]"
-            )
-    else:
-        from airflow.utils.serve_logs import serve_logs  # type: ignore[no-redef]
+    from airflow.utils.serve_logs import serve_logs
path: providers/celery/src/airflow/providers/celery/cli/celery_command.py
user: login: kaxil
body: Its there: https://github.com/apache/airflow/blob/2.9.3/airflow/utils/serve_logs.py#L168-L183 
created_at: 2025-04-10T08:35:42Z
author_association: MEMBER

PR comments:
user: login: amoghrajesh
created_at: 2025-04-10T06:12:52Z
body: A small static check failure: 
```
diff --git a/providers/celery/README.rst b/providers/celery/README.rst
index d2c6bfc..73789e8 100644
--- a/providers/celery/README.rst
+++ b/providers/celery/README.rst
@@ -75,7 +75,6 @@ You can install such cross-provider dependencies when installing from PyPI. For
 Dependent package                                                                                                       Extra
 ======================================================================================================================  ===================
 `apache-airflow-providers-cncf-kubernetes <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes>`_  ``cncf.kubernetes``
-`apache-airflow-providers-fab <https://airflow.apache.org/docs/apache-airflow-providers-fab>`_                          ``fab``
 ======================================================================================================================  ===================
 
 The changelog for the provider package can be found in the
diff --git a/providers/celery/pyproject.toml b/providers/celery/pyproject.toml
index 1f002d0..93feb1c 100644
--- a/providers/celery/pyproject.toml
+++ b/providers/celery/pyproject.toml
@@ -82,7 +82,6 @@ dev = [
     "apache-airflow-task-sdk",
     "apache-airflow-devel-common",
     "apache-airflow-providers-cncf-kubernetes",
-    "apache-airflow-providers-fab",
     # Additional devel dependencies (do not remove this line and add extra development dependencies)
 ]

```
author_association: CONTRIBUTOR
user: login: potiuk
created_at: 2025-04-10T13:25:02Z
body: Thanks @kaxil for the fixup :D
author_association: MEMBER
user: login: vincbeck
created_at: 2025-04-10T13:29:53Z
body: Thanks!
author_association: CONTRIBUTOR