Repository: apache/airflow

Issue:
CLI `airflow scheduler -D --pid <PIDFile>` fails silently if PIDFile given is a relative path


<!--

Welcome to Apache Airflow!  For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)

If you need to include code snippets or logs, please put them in fenced code
blocks.  If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>

Please delete these comment blocks before submitting the issue.

-->

<!--

IMPORTANT!!!

PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!

PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!

Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.

-->

**Apache Airflow version**: 2.0.0

**Environment**: Linux & MacOS, venv

- **OS** (e.g. from /etc/os-release): Ubuntu 18.04.3 LTS / MacOS 10.15.7 
- **Kernel** (e.g. `uname -a`):
  - Linux *** 5.4.0-1029-aws #30~18.04.1-Ubuntu SMP Tue Oct 20 11:09:25 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
  - Darwin *** 19.6.0 Darwin Kernel Version 19.6.0: Thu Oct 29 22:56:45 PDT 2020; root:xnu-6153.141.2.2~1/RELEASE_X86_64 x86_64

**What happened**:

Say I'm in my home dir, running command `airflow scheduler -D --pid test.pid` (`test.pid` is a relative path) is supposed to start the scheduler in daemon mode, and the PID will be stored in the file `test.pid` (if it doesn't exist, it should be created).

However, the scheduler is NOT started. This can be validated by running `ps aux | grep airflow | grep scheduler` (no process is shown). In the whole process, I don't see any error message.

However, if I change the pid file path to an absolute path, i.e. `airflow scheduler -D --pid ${PWD}/test.pid`, it successfully start the scheduler in daemon mode (can be validated via the method above).

**What you expected to happen**:

Even if the PID file path provided is a relative path, the scheduler should be started properly as well.

<!-- What do you think went wrong? -->

**How to reproduce it**:

Described above

<!---

As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.

If you are using kubernetes, please attempt to recreate the issue using minikube or kind.

## Install minikube/kind

- Minikube https://minikube.sigs.k8s.io/docs/start/
- Kind https://kind.sigs.k8s.io/docs/user/quick-start/

If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action

You can include images using the .md style of
![alt text](http://url/to/img.png)

To record a screencast, mac users can use QuickTime and then create an unlisted youtube video with the resulting .mov file.

--->


**Anything else we need to know**:

<!--

How often does this problem occur? Once? Every time etc?

Any relevant logs to include? Put them here in side a detail tag:
<details><summary>x.log</summary> lots of stuff </details>

-->



Discussion:
Fixing this issue is straigth-forward: when a PID file path is provided, always extend it to a full absolute path first. But I would like to have someone to help confirm they can reproduce the error first, meanwhile I will find out the root-cause as well of course.
Confirmed. 


The reason is that default "working_directory" is used in  in DaemonContext https://github.com/apache/airflow/blob/97eee350e41f6f00ad3076489a5dac2e5beaa266/airflow/cli/commands/scheduler_command.py#L48 

By default the "working_directory" is set to "/" which means that the pid file is attempted to be created as "/<FILE>" - which for non-root user will fail. This works for example in Breeze as we use root user there, but in production setup with non-root user it will fail. 

I think we should assume (extending to the code in https://github.com/apache/airflow/blob/97eee350e41f6f00ad3076489a5dac2e5beaa266/airflow/utils/cli.py#L222) than when relative path is specified, it is relative to ${AIRFLOW_HOME}.
Thanks @potiuk for helping identify the root cause.

I have different opinion: if it's a relative path, it should be relative to current directory where the user is running the command. I.e., as I mentioned above, if it's relative path X, extend it to CURRENT_DIR/X (rather than assuming it to be AORFLOW_HOME)

Let me know if you agree?
I think when you deamonize a process, you actually want to run it in a "specific" directory. The whole idea about running process as a daemon in *NIX is to detach it completely from the originating process/command (that's why for example if you want to make a really detached daemon you should fork it twice https://stackoverflow.com/questions/881388/what-is-the-reason-for-performing-a-double-fork-when-creating-a-daemon to avoid zombies (making sure that the daemon process is parented by init process). That's why also the "DaemonContext' has the "/" as working directory set. This is not a mistake - it is pretty deliberate behaviour. 

For example such daemonized process should survive and work if someone decides do delete the whole "current directory" whatever it was. The current working directory is pretty much accidental, so creating an output of "daemon" there is not really what I'd expect.
I'm more thinking from an "expectation" perspective:

Say I'm in directory "/home/xd" (my AIRFLOW_HOME may be `/home/airflow_user/airflow`), and I run `airflow scheduler -D --pid test.pid`. When I specify `test.pid`, I'm expecting the PID to be saved in this PID file in current directory, whose path is `/home/xd/test.pid`. I don't expect the full path to be `/home/airflow_user/airflow/test.pid`, which will be a confusion.

On the other hand, when we daemonize this process, the only thing to do with path/directory is this file where PID is stored.  We don't run a process in a "specific" directory.
So maybe it's better to require a non-relative file and fail with error if it is not ? 
Haha, I agree that's a "compromise" which can make both of us happy. Why not😉

I will create a PR shortly tonight for this.
> The current working directory is pretty much accidental, so creating an output of "daemon" there is not really what I'd expect.

I view it as the opposite. Every* unix program that deals with filenames on the command line treats a non-absolute path input as relative to the current directory. We should behave the same. (The fact the program is deamonized is an implementation detail.)

\* every one I can think of 
I think we should take https://linux.die.net/man/1/daemonize as the canonical behaviour. 

It supports both - changing the working directory (default as "/") and specifying pidfile at the command line. It is not really specified in the man page what happens if you specify relative path, but maybe we can check and do the same.
But for our case, it may not be a good idea to further expose "working directory" or allow changing it. Making it simple should be better.

I prefer to my very first proposal (if the path is relative, expand it with the current directory) which aligns with @ashb 's suggestion. Meanwhile I'm open to different solution if we come to an agreement by majority.
I am not sure we should reinvent the wheel. I just wonder what "deamonize" does when you do not specify the working dir. I think we should do exactly the same. Not sure how relative paths are treated by it, but I believe we should do exactly what it does.
What deamonize does is not relevant for the behaviour we want.

We are accepting a file input from the command line: what is the most common pattern, and therefor least friction/most expected to a user.
I start to doubt if we are discussing on the same thing now :-)
Since you asked though - daemonize(1) creates the pid file, then changes directory -- i.e. relative pid file would be relative to the cwd.


https://github.com/bmc/daemonize/blob/master/daemonize.c#L461-L500
Hi guys @ashb @potiuk shall we come to a consensus on this?
No strong  opinion there :). If you want to use relative from cwd, I am fine with it :).
> No strong  opinion there :). If you want to use relative from cwd, I am fine with it :).

Yes please - let's do that then @XD-DENG 
Thanks both @potiuk  @ashb . The discussion itself was thorough and concrete.

I will update PR https://github.com/apache/airflow/pull/13232 accordingly.

Resolving patch:
diff --git a/airflow/utils/cli.py b/airflow/utils/cli.py
--- a/airflow/utils/cli.py
+++ b/airflow/utils/cli.py
@@ -227,8 +227,11 @@ def setup_locations(process, pid=None, stdout=None, stderr=None, log=None):
         stdout = os.path.join(settings.AIRFLOW_HOME, f'airflow-{process}.out')
     if not log:
         log = os.path.join(settings.AIRFLOW_HOME, f'airflow-{process}.log')
+
     if not pid:
         pid = os.path.join(settings.AIRFLOW_HOME, f'airflow-{process}.pid')
+    else:
+        pid = os.path.abspath(pid)
 
     return pid, stdout, stderr, log
 


Test patch:
diff --git a/tests/utils/test_cli_util.py b/tests/utils/test_cli_util.py
--- a/tests/utils/test_cli_util.py
+++ b/tests/utils/test_cli_util.py
@@ -131,6 +131,23 @@ def test_cli_create_user_supplied_password_is_masked(self, given_command, expect
         command = json.loads(command.replace("'", '"'))
         self.assertEqual(command, expected_command)
 
+    def test_setup_locations_relative_pid_path(self):
+        relative_pid_path = "fake.pid"
+        pid_full_path = os.path.join(os.getcwd(), relative_pid_path)
+        pid, _, _, _ = cli.setup_locations(process="fake_process", pid=relative_pid_path)
+        self.assertEqual(pid, pid_full_path)
+
+    def test_setup_locations_absolute_pid_path(self):
+        abs_pid_path = os.path.join(os.getcwd(), "fake.pid")
+        pid, _, _, _ = cli.setup_locations(process="fake_process", pid=abs_pid_path)
+        self.assertEqual(pid, abs_pid_path)
+
+    def test_setup_locations_none_pid_path(self):
+        process_name = "fake_process"
+        default_pid_path = os.path.join(settings.AIRFLOW_HOME, f"airflow-{process_name}.pid")
+        pid, _, _, _ = cli.setup_locations(process=process_name)
+        self.assertEqual(pid, default_pid_path)
+
 
 @contextmanager
 def fail_action_logger_callback():