Repository: apache/airflow

Issue:
Airflow 2.7.1 can not start Scheduler & trigger
### Apache Airflow version

2.7.1

### What happened

After upgrade from 2.6.0 to 2.7.1 (try pip uninstall apache-airflow, and clear dir airflow - remove airflow.cfg), I can start scheduler & trigger with daemon. 
I try start with command, it can start, but logout console it killed.
I try: airflow scheduler or airflow triggerer :done but kill when logout console
airflow scheduler --daemon && airflow triggerer --daemon: fail, can not start scheduler & triggerer (2.6.0 run ok). but start deamon with webserver & celery worker is fine

Help me

### What you think should happen instead

_No response_

### How to reproduce

1. run airflow 2.6.0 fine on ubuntu server 22.04.3 lts
2. install airflow 2.7.1 
3. can not start daemon triggerer & scheduler

### Operating System

ubuntu server 22.04.3 LTS

### Versions of Apache Airflow Providers

_No response_

### Deployment

Virtualenv installation

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

Some questions regarding your report:

- Have you tried a clean install of Airflow 2.7.1 as well?
- Are you running in a Venv or in a system site package install?
- Can you please attach some logs to the problem? Especially what the logs of daemon processes are?
- Also can you describe how your service is usually instantiated? If you kill the terminal where you started it is normal that child processes are terminated as well. If you run in daemon mode they usually stay alive.

- i run venv
- if start airflow triggerer then run and show log to console, but ctrl+c or close terminal triggerer killed.
- if start airflow trigger --daemon, i dont show any log, i dont known
airflow triggerer --capacity 5 --log-file ./logs/triggerer/triggerer.log --pid ./triggerer.pid --daemon
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2023-10-10T00:03:58.394+0700] {triggerer_job_runner.py:171} INFO - Setting up TriggererHandlerWrapper with handler <FileTaskHandler (NOTSET)>
[2023-10-10T00:03:58.397+0700] {triggerer_job_runner.py:227} INFO - Setting up logging queue listener with handlers [<RedirectStdHandler <stdout> (NOTSET)>, <TriggererHandlerWrapper (NOTSET)>]

--> any log on console, not show log in file
Sorry I can not understand you last comment completely.

- Can you please repeat and also add the verbose option via `--verbose` to the command?
- And can you also please post the logs of the scheduler as well?
- Have you tried a clean install of Airflow 2.7.1 as well?
Hi, 
I try:
Install new server (ubuntu server 22.04.3 LTS) and postgresql 16, 
Install airflow:

wget https://raw.githubusercontent.com/apache/airflow/constraints-2.7.1/constraints-3.10.txt
pip3 install "apache-airflow[celery,cgroups,dask,statsd,virtualenv,redis]==2.7.1" --constraint "/home/constraints-3.10.txt"

then:
- config AIRFLOW_HOME in ~/.profile
-  set up database on postgres and run airflow db migrate
- config database on airflow.cfg
- run: 
    +/ airflow webserver --daemon --> ok, show webui on brower
    +/ airflow scheduler --daemon --> ok, show on cluster activity tab --> scheduler was Healthy
    +/ airflow triggerer --daemon --> NOT OK, show on cluter actity tab --> scheduler was Healthy, TRIGGERER UNHEALTHY
i try:
  - airflow trigger --> cluster activity tab --> scheduler & triggerer was healthy but close terminal triggerer killed.

airflow triggerer --daemon, it's NOT OK:
<img width="1351" alt="Screenshot 2023-10-11 at 12 25 50" src="https://github.com/apache/airflow/assets/95118800/fb69ad44-0bad-428e-8d60-898e38770e4c">

<img width="556" alt="Screenshot 2023-10-11 at 12 29 54" src="https://github.com/apache/airflow/assets/95118800/6601a7ee-b330-4119-a326-1e439dcfa903">

airflow triggerer (without --daemon), it's OK:
<img width="1368" alt="Screenshot 2023-10-11 at 12 26 35" src="https://github.com/apache/airflow/assets/95118800/c5a85f00-f7b1-4638-be2e-a4254fc9487c">

run with -v (error same without -v), it's NOT OK
<img width="1440" alt="Screenshot 2023-10-11 at 12 36 04" src="https://github.com/apache/airflow/assets/95118800/6209f4bc-7e93-428f-b4d4-99f2413b4729">
<img width="1434" alt="Screenshot 2023-10-11 at 12 36 18" src="https://github.com/apache/airflow/assets/95118800/ed3c8a25-a04d-4b7f-ace6-ddd24b627174">

I think if run triggerer on daemon, it not start gunicorn (i show gunicorn start on log file). if i run without daemon, i show starting gunicorn on console.
The same issue with 2.7.2, `airflow triggerer -D` not working anymore. In the error log file, i see that triggerer was kill by a `term` signal `Handling signal: term` right after worker booted.

``` text
[2023-10-13 12:15:23 +0800] [722477] [INFO] Starting gunicorn 21.2.0
[2023-10-13 12:15:23 +0800] [722477] [INFO] Listening at: http://[::]:8794 (722477)
[2023-10-13 12:15:23 +0800] [722477] [INFO] Using worker: sync
[2023-10-13 12:15:23 +0800] [722479] [INFO] Booting worker with pid: 722479
[2023-10-13 12:15:23 +0800] [722480] [INFO] Booting worker with pid: 722480
[2023-10-13 12:15:23 +0800] [722477] [INFO] Handling signal: term
[2023-10-13 12:15:23 +0800] [722479] [INFO] Worker exiting (pid: 722479)
[2023-10-13 12:15:23 +0800] [722480] [INFO] Worker exiting (pid: 722480)
[2023-10-13 12:15:23 +0800] [722477] [INFO] Shutting down: Master
```



I tested a couple of versions and can confirm it, the regression took place between 2.6.2 and 2.6.3 - in 2.6.3 (and later), after starting triggerer with -D option, `airflow-triggerer.pid` is not created, the process dies immediately.
> I tested a couple of versions and can confirm it, the regression took place between 2.6.2 and 2.6.3 - in 2.6.3 (and later), after starting triggerer with -D option, `airflow-triggerer.pid` is not created, the process dies immediately.

maybe can fix it or wait new version of airflow?
I think I isolated the code change that caused the problem - the problem starts to appear after this commit: https://github.com/apache/airflow/commit/d6cc9e4bb1efe9713eccd8e62e46f11bad294a36#diff-9cea7921268261a177e82c16fd5111f8d3252e3ca0267bdfb397c379c5d70857R353-R355

When I comment out these lines, the triggerer starts and the demon thread continues to run successfully.

@ThuanDoHung for now I'm trying to triage the bug, don't know how to fix it yet
After debugging I think I understand the problem fully. Here is what happens when we run triggerer in daemon mode on current main.

1. Triggerer's main thread creates triggerer_job_runner [triggerer_job_runner](https://github.com/apache/airflow/blob/e9987d50598f70d84cbb2a5d964e21020e81c080/airflow/cli/commands/triggerer_command.py#L61C5-L61C25) which internally creates TriggerRunner - an async thread, but doesn't start it yet.
2. Triggerer's main thread enters daemon context https://github.com/apache/airflow/blob/e9987d50598f70d84cbb2a5d964e21020e81c080/airflow/cli/commands/triggerer_command.py#L72 which internally forks itself and exits in parent process so that only fork continues to run https://pagure.io/python-daemon/blob/main/f/daemon/daemon.py#_683
3. After fork only the calling thread stays alive as it is the result of POSIX complaince. Python respects it by setting all other threads to `_is_stopped = True` https://github.com/python/cpython/blob/b2ab210aaefb3b0e39f28e7946b7a531d7b2ab17/Lib/threading.py#L1690 This affects the TriggerRunner thread which is set to stopped as well.
4.  TriggerRunner is started successfully https://github.com/apache/airflow/blob/e9987d50598f70d84cbb2a5d964e21020e81c080/airflow/jobs/triggerer_job_runner.py#L337 Apparently being in a stopped state doesn't prevent it from starting.
5. By the time when the code reaches the `is_alive()` check it returns false (because is_alive() inspects internal state where `_is_stopped = True`) so the code finishes execution early. https://github.com/apache/airflow/commit/d6cc9e4bb1efe9713eccd8e62e46f11bad294a36#diff-9cea7921268261a177e82c16fd5111f8d3252e3ca0267bdfb397c379c5d70857R353-R355 

Before the change https://github.com/apache/airflow/pull/32092 it wasn't a problem because apparently a thread in stopped state runs fine unless someone verifies its state (which is inconsistent).

My proposal to fix it: 
1) Switch order of operations when running triggerer command in daemon mode so that the async thread is created after entering daemon context, e.g. move the thread initialization https://github.com/apache/airflow/blob/e9987d50598f70d84cbb2a5d964e21020e81c080/airflow/cli/commands/triggerer_command.py#L61C5-L61C25 from line  61 to 80 and 86 (worked when I tested it locally).
2) Refactor all the commands that have daemon mode (scheduler, webserver, etc.) to reuse the same pattern.

@jens-scheffler-bosch I'm willing to work on it
Wow. Cool analysis. Respect. This is not an easy catch! Thanks for investing the time locating the root!

Yes, with this explanaition it all makes much sense. Happy to see a contribution!

Resolving patch:
diff --git a/airflow/cli/commands/triggerer_command.py b/airflow/cli/commands/triggerer_command.py
--- a/airflow/cli/commands/triggerer_command.py
+++ b/airflow/cli/commands/triggerer_command.py
@@ -58,7 +58,6 @@ def triggerer(args):
     settings.MASK_SECRETS_IN_LOGS = True
     print(settings.HEADER)
     triggerer_heartrate = conf.getfloat("triggerer", "JOB_HEARTBEAT_SEC")
-    triggerer_job_runner = TriggererJobRunner(job=Job(heartrate=triggerer_heartrate), capacity=args.capacity)
 
     if args.daemon:
         pid, stdout, stderr, log_file = setup_locations(
@@ -77,10 +76,16 @@ def triggerer(args):
                 umask=int(settings.DAEMON_UMASK, 8),
             )
             with daemon_context, _serve_logs(args.skip_serve_logs):
+                triggerer_job_runner = TriggererJobRunner(
+                    job=Job(heartrate=triggerer_heartrate), capacity=args.capacity
+                )
                 run_job(job=triggerer_job_runner.job, execute_callable=triggerer_job_runner._execute)
     else:
         signal.signal(signal.SIGINT, sigint_handler)
         signal.signal(signal.SIGTERM, sigint_handler)
         signal.signal(signal.SIGQUIT, sigquit_handler)
         with _serve_logs(args.skip_serve_logs):
+            triggerer_job_runner = TriggererJobRunner(
+                job=Job(heartrate=triggerer_heartrate), capacity=args.capacity
+            )
             run_job(job=triggerer_job_runner.job, execute_callable=triggerer_job_runner._execute)


Test patch: