Repository: apache/airflow

Issue:
Allow Dag Processor Manager to log to stdout
### Description

On Airflow 2.8.1, the DAG processor manager no longer logs out to stdout (whereas it did on 2.6.2).

### Use case/motivation

We currently have a configuration option for the DAG processor to log to stdout via `AIRFLOW__LOGGING__DAG_PROCESSOR_LOG_TARGET`. A similar configuration `AIRFLOW__LOGGING__DAG_PROCESSOR_MANAGER_LOG_TARGET` could achieve that.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

afaik, the DAG processor manager runs in a spawned process and by default output its log to a destinated file defined by `dag_processor_manager_log_location`. I think this behaviour also applies to `v2.6.2`

I suggest adding an extra variable in the config like `dag_processor_manager_log_stdout` , which if being set to `True` (default is `False`) , extend the log handler to also include logging to stdout. That way, the log from the spawned process can also be directed to the stdout of airflow scheduler like below : 

<img width="1399" alt="image" src="https://github.com/apache/airflow/assets/43698890/645fe926-ed81-4766-80c0-e0a3fa673155">


thoughts ?

> thoughts ?

 This is a valid option if someone wants it, all it needs is someone to PR it.
> > thoughts ?
> 
> This is a valid option if someone wants it, all it needs is someone to PR it.

Ok I'm on it

Resolving patch:
diff --git a/airflow/config_templates/airflow_local_settings.py b/airflow/config_templates/airflow_local_settings.py
--- a/airflow/config_templates/airflow_local_settings.py
+++ b/airflow/config_templates/airflow_local_settings.py
@@ -56,6 +56,10 @@
     "logging", "DAG_PROCESSOR_MANAGER_LOG_LOCATION"
 )
 
+DAG_PROCESSOR_MANAGER_LOG_STDOUT: str = conf.get_mandatory_value(
+    "logging", "DAG_PROCESSOR_MANAGER_LOG_STDOUT"
+)
+
 # FILENAME_TEMPLATE only uses in Remote Logging Handlers since Airflow 2.3.3
 # All of these handlers inherited from FileTaskHandler and providing any value rather than None
 # would raise deprecation warning.
@@ -171,6 +175,19 @@
     },
 }
 
+if DAG_PROCESSOR_MANAGER_LOG_STDOUT == "True":
+    DEFAULT_DAG_PARSING_LOGGING_CONFIG["handlers"].update(
+        {
+            "console": {
+                "class": "airflow.utils.log.logging_mixin.RedirectStdHandler",
+                "formatter": "airflow",
+                "stream": "sys.stdout",
+                "filters": ["mask_secrets"],
+            }
+        }
+    )
+    DEFAULT_DAG_PARSING_LOGGING_CONFIG["loggers"]["airflow.processor_manager"]["handlers"].append("console")
+
 # Only update the handlers and loggers when CONFIG_PROCESSOR_MANAGER_LOGGER is set.
 # This is to avoid exceptions when initializing RotatingFileHandler multiple times
 # in multiple processes.


Test patch:
diff --git a/tests/dag_processing/test_job_runner.py b/tests/dag_processing/test_job_runner.py
--- a/tests/dag_processing/test_job_runner.py
+++ b/tests/dag_processing/test_job_runner.py
@@ -1632,3 +1632,37 @@ def test_heartbeat_manager_end_no_process(self):
         processor_agent.end()
         processor_agent.log.warning.assert_called_with("Ending without manager process.")
         processor_agent._process.join.assert_not_called()
+
+    @conf_vars({("logging", "dag_processor_manager_log_stdout"): "True"})
+    def test_log_to_stdout(self, capfd):
+        test_dag_path = TEST_DAG_FOLDER / "test_scheduler_dags.py"
+        async_mode = "sqlite" not in conf.get("database", "sql_alchemy_conn")
+
+        # Starting dag processing with 0 max_runs to avoid redundant operations.
+        processor_agent = DagFileProcessorAgent(test_dag_path, 0, timedelta(days=365), [], False, async_mode)
+        processor_agent.start()
+        if not async_mode:
+            processor_agent.run_single_parsing_loop()
+
+        processor_agent._process.join()
+
+        # Capture the stdout and stderr
+        out, _ = capfd.readouterr()
+        assert "DAG File Processing Stats" in out
+
+    @conf_vars({("logging", "dag_processor_manager_log_stdout"): "False"})
+    def test_not_log_to_stdout(self, capfd):
+        test_dag_path = TEST_DAG_FOLDER / "test_scheduler_dags.py"
+        async_mode = "sqlite" not in conf.get("database", "sql_alchemy_conn")
+
+        # Starting dag processing with 0 max_runs to avoid redundant operations.
+        processor_agent = DagFileProcessorAgent(test_dag_path, 0, timedelta(days=365), [], False, async_mode)
+        processor_agent.start()
+        if not async_mode:
+            processor_agent.run_single_parsing_loop()
+
+        processor_agent._process.join()
+
+        # Capture the stdout and stderr
+        out, _ = capfd.readouterr()
+        assert "DAG File Processing Stats" not in out