Repository: pantsbuild/pants

Issue:
airflow pex fails to find the embedded gunicorn binary 
_(From Mingshi Wang on Slack)_

The context is I packed Airflow into a PEX file which depends on "gunicorn".  I have verified that the Airflow PEX file contains "gunicorn", which is perfect. However, the path of "gunicorn" inside the unzipped PEX file (a folder under /tmp) is not known before hand, and hence when Airflow launches gunicorn as a subprocess, it errors out immediately. I'd like to know if there is a way to specify that target dir that the PEX file is unzipped into  when being executed? If this is feasible, then the path to "gunicorn" is known beforehand and can be saved to $PATH env, so it airflow can launch it as a subprocess. 

See this document for details:

https://docs.google.com/document/d/1bKjy5j88NIWHC8xTOHL6jYD0pmU4lKEmsdyph9xVqxk/edit#


Discussion:
If I build the airflow pex like this:

```
pex apache-airflow[async,crypto,celery,kubernetes,jdbc,password,postgres,s3,slack,amazon] -o airflow.pex
```

And then run steps as described here: https://airflow.apache.org/docs/apache-airflow/stable/start/local.html 

```
airflow db init

airflow users create \
    --username admin \
    --firstname Peter \
    --lastname Parker \
    --role Admin \
    --email spiderman@superhero.org

airflow webserver --port 8080
```

Then gunicorn does start up.

So something may be different about the airflow pex the user has built.
1st you must not have gunicorn on your `PATH` to repro:
```console
$ which gunicorn
which: no gunicorn in (/home/jsirois/.rvm/gems/ruby-2.6.0/bin:/home/jsirois/.rvm/gems/ruby-2.6.0@global/bin:/home/jsirois/.rvm/rubies/ruby-2.6.0/bin:/home/jsirois/.cargo/bin:/home/jsirois/.pyenv/shims:/home/jsirois/bin:/home/jsirois/.local/bin:/home/jsirois/.nvm/versions/node/v8.11.3/bin:/home/jsirois/.poetry/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/opt/cuda/bin:/opt/cuda/nsight_compute:/opt/cuda/nsight_systems/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/jsirois/.bash.d/bin:/home/jsirois/go/bin:/home/jsirois/.gem/ruby/2.5.0/bin:/home/jsirois/.rvm/bin:/home/jsirois/.bash.d/bin)
```

Next, proceeding down @benjyw's attempt:
```console
$ pex apache-airflow[async,crypto,celery,kubernetes,jdbc,password,postgres,s3,slack,amazon] -o airflow.pex
Failed to resolve compatible distributions:
1: Jinja2==3.0.1 requires MarkupSafe>=2.0 but MarkupSafe 1.1.1 was resolved
2: Flask-AppBuilder==3.3.2 requires SQLAlchemy<1.4.0 but SQLAlchemy 1.4.23 was resolved
3: Flask==1.1.4 requires Jinja2<3.0,>=2.10.1 but Jinja2 3.0.1 was resolved
4: Flask==1.1.4 requires click<8.0,>=5.1 but click 8.0.1 was resolved
```
So this needs to use the new Pip resolver maybe?:
```console
$ pex --resolver-version pip-2020-resolver apache-airflow[async,crypto,celery,kubernetes,jdbc,password,postgres,s3,slack,amazon] -o airflow.pex
$ ./airflow.pex db init
Could not open db in the environment [./airflow.pex]: [Errno 2] No such file or directory: 'db'
$
```
Ah, forgot the entry point:
```console
$ pex --resolver-version pip-2020-resolver apache-airflow[async,crypto,celery,kubernetes,jdbc,password,postgres,s3,slack,amazon] -o airflow.pex -c airflow
$ ./airflow.pex db init
DB: sqlite:////home/jsirois/airflow/airflow.db
[2021-08-25 11:02:05,620] {db.py:702} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
...
INFO  [airflow.models.dag] Sync 2 DAGs
INFO  [airflow.models.dag] Setting next_dagrun for example_subdag_operator.section-1 to None
INFO  [airflow.models.dag] Setting next_dagrun for example_subdag_operator.section-2 to None
Initialization done
$ $ ./airflow.pex users create \
    --username admin \
    --firstname Peter \
    --lastname Parker \
    --role Admin \
    --email spiderman@superhero.org
[2021-08-25 11:02:26,912] {manager.py:788} WARNING - No user yet created, use flask fab command to do it.
Password:
Repeat for confirmation:
Admin user admin created
$ ./airflow.pex webserver --port 8080
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2021-08-25 11:02:51,241] {dagbag.py:496} INFO - Filling up the DagBag from /dev/null
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
Access Logformat: 
=================================================================            
Traceback (most recent call last):
  File "/home/jsirois/dev/pantsbuild/jsirois-pex/airflow.pex/.bootstrap/pex/pex.py", line 483, in execute
  File "/home/jsirois/dev/pantsbuild/jsirois-pex/airflow.pex/.bootstrap/pex/pex.py", line 401, in _wrap_coverage
  File "/home/jsirois/dev/pantsbuild/jsirois-pex/airflow.pex/.bootstrap/pex/pex.py", line 432, in _wrap_profiling
  File "/home/jsirois/dev/pantsbuild/jsirois-pex/airflow.pex/.bootstrap/pex/pex.py", line 539, in _execute
  File "/home/jsirois/dev/pantsbuild/jsirois-pex/airflow.pex/.bootstrap/pex/pex.py", line 655, in execute_entry
  File "/home/jsirois/dev/pantsbuild/jsirois-pex/airflow.pex/.bootstrap/pex/pex.py", line 687, in execute_pkg_resources
  File "/home/jsirois/.pex/installed_wheels/eeea2b3eed91fe2b349bbea1bd75160a0783e634/apache_airflow-2.1.3-py3-none-any.whl/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/home/jsirois/.pex/installed_wheels/eeea2b3eed91fe2b349bbea1bd75160a0783e634/apache_airflow-2.1.3-py3-none-any.whl/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/home/jsirois/.pex/installed_wheels/eeea2b3eed91fe2b349bbea1bd75160a0783e634/apache_airflow-2.1.3-py3-none-any.whl/airflow/utils/cli.py", line 91, in wrapper
    return f(*args, **kwargs)
  File "/home/jsirois/.pex/installed_wheels/eeea2b3eed91fe2b349bbea1bd75160a0783e634/apache_airflow-2.1.3-py3-none-any.whl/airflow/cli/commands/webserver_command.py", line 483, in webserver
    with subprocess.Popen(run_args, close_fds=True) as gunicorn_master_proc:
  File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'gunicorn'
$
```

So a repro. But `--venv` mode supports adding all venv `bin/` scripts to the path, so let's try that:
```console
$ pex --resolver-version pip-2020-resolver apache-airflow[async,crypto,celery,kubernetes,jdbc,password,postgres,s3,slack,amazon] -o airflow.pex -c airflow --venv append
$ ./airflow.pex webserver --port 8080
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2021-08-25 11:07:38,149] {dagbag.py:496} INFO - Filling up the DagBag from /dev/null
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
Access Logformat: 
=================================================================            
[2021-08-25 11:07:39 -0700] [224568] [INFO] Starting gunicorn 20.1.0
[2021-08-25 11:07:39 -0700] [224568] [INFO] Listening at: http://0.0.0.0:8080 (224568)
[2021-08-25 11:07:39 -0700] [224568] [INFO] Using worker: sync
[2021-08-25 11:07:39 -0700] [224569] [INFO] Booting worker with pid: 224569
[2021-08-25 11:07:39 -0700] [224570] [INFO] Booting worker with pid: 224570
[2021-08-25 11:07:40 -0700] [224571] [INFO] Booting worker with pid: 224571
[2021-08-25 11:07:40 -0700] [224579] [INFO] Booting worker with pid: 224579
```

Success.

So Mingshi just needs to use `--venv append` (or `--venv prepend`). Unfortunately, Pants doesn't yet support this. It only supports `--venv`:
https://github.com/pantsbuild/pants/blob/d4cf50ea67abe7019835506f37994494f723a74a/src/python/pants/backend/python/goals/package_pex_binary.py#L90-L91


Re-reading the OP: Was this reported as a Pants bug @benjyw or a Pex bug? IOW: Is a feature in Pants actually needed to unblock Mingshi or are they using Pex directly and so can be unblocked by just adding `--venv append` to their command line?
I believe they are using Pants to build the airflow pex.
Huh, I can now reproduce it. Maybe the thing I was missing when I had it working was that gunicorn was on my PATH for other reasons. 
> I believe they are using Pants to build the airflow pex.

Ah yes - sorry. That's right there in the linked doc I failed to read through.

I think you guys nailed it, but fwiw, made a repo easier for repro based on pants and docker
https://github.com/jinghanxu1-coinbase/example-airflow-pants-pex

we are using pants to build pex, and like ^ mentioned, the pex doesn't have its dependencies' bin in PATH. But sounds like it's configurable from pex side, so likely we will just need pants to expose that option

However, assuming it could find the gunicorn bin, I'm also surprise it would work right after. The gunicorn spins off by subprocess.popen from airflow process, and based on my experiment, subprocess won't carry on the sys.path from parent process, which means, all the dependencies packed in pex won't be reachable from gunicorn, and it seems needed
> However, assuming it could find the gunicorn bin, I'm also surprise it would work right after. The gunicorn spins off by subprocess.popen from airflow process, and based on my experiment, subprocess won't carry on the sys.path from parent process, which means, all the dependencies packed in pex won't be reachable from gunicorn, and it seems needed

All bin python scripts in Pex `--venv` mode have their python shebang lines re-written to point to the PEX venv interpreter for this very reason. So gunicorn will see all your PEX dependencies just fine.
So what remains is to add support for this in Pants. Do we add another pex mode, or do we just always do `--venv append` in `venv` mode?
I think having a build option for it, with value of `append(default) | not_append` ( prob a better name ) is good. But up to you guys 
I think defaulting to `--venv prepend` is probably fine. This is what a venv does when you source its `bin/activate` script and it avoids doing the wrong thing if you have `gunicorn` already on your PATH in append mode (the OS will pick the wrong gunicorn which will not have your dependencies on its `sys.path`). I'll post a PR with that change and it will be available in Friday's dev release (2.7.0.dev5 but we may be cutting 2.7.0rc0 - not sure).

Resolving patch:
diff --git a/src/python/pants/backend/python/goals/package_pex_binary.py b/src/python/pants/backend/python/goals/package_pex_binary.py
--- a/src/python/pants/backend/python/goals/package_pex_binary.py
+++ b/src/python/pants/backend/python/goals/package_pex_binary.py
@@ -88,7 +88,7 @@ def generate_additional_args(self, pex_binary_defaults: PexBinaryDefaults) -> Tu
         if self._execution_mode is PexExecutionMode.UNZIP:
             args.append("--unzip")
         if self._execution_mode is PexExecutionMode.VENV:
-            args.append("--venv")
+            args.extend(("--venv", "prepend"))
         if self.include_tools.value is True:
             args.append("--include-tools")
         return tuple(args)


Test patch:
diff --git a/src/python/pants/backend/python/goals/run_pex_binary_integration_test.py b/src/python/pants/backend/python/goals/run_pex_binary_integration_test.py
--- a/src/python/pants/backend/python/goals/run_pex_binary_integration_test.py
+++ b/src/python/pants/backend/python/goals/run_pex_binary_integration_test.py
@@ -90,6 +90,9 @@ def run(*extra_args: str, **extra_env: str) -> PantsResult:
         pex_info = json.loads(result.stdout)
         assert (execution_mode is PexExecutionMode.UNZIP) == pex_info["unzip"]
         assert (execution_mode is PexExecutionMode.VENV) == pex_info["venv"]
+        assert ("prepend" if execution_mode is PexExecutionMode.VENV else "false") == pex_info[
+            "venv_bin_path"
+        ]
         assert pex_info["strip_pex_env"] is False