Repository: wandb/wandb

Issue:
[CLI]: Creating a wandb server doesn't work with Pex/PyTorch Lightning
### Describe the bug

<!--- Description of the issue below  -->
When trying to use the pytorch-lightning integrations with Weights and Biases, my code fails to run when using Pex as a packaging service. This is what's used in the popular Pants project. It appears to be an issue with how Pex runs programs in a sandbox, and the wandb library is trying to execute a subprocess in this sandbox somehow without permission.

<!--- A minimal code snippet between the quotes below  -->
I have reproduced a minimal example in this Github repo

https://github.com/nharada1/pex-wandb-repro

<!--- A full traceback of the exception in the quotes below -->
```shell
Traceback (most recent call last):
  File "/home/user/.cache/pants/named_caches/pex_root/venvs/e33dbad2e381237c2ef87dd830f8fbcc73be777f/7898379b30005666861db5e178f56a2477d285aa/lib/python3.9/site-packages/wandb/sdk/wandb_init.py", line 1075, in i
nit
    wi.setup(kwargs)
  File "/home/user/.cache/pants/named_caches/pex_root/venvs/e33dbad2e381237c2ef87dd830f8fbcc73be777f/7898379b30005666861db5e178f56a2477d285aa/lib/python3.9/site-packages/wandb/sdk/wandb_init.py", line 165, in se
tup
    self._wl = wandb_setup.setup()
  File "/home/user/.cache/pants/named_caches/pex_root/venvs/e33dbad2e381237c2ef87dd830f8fbcc73be777f/7898379b30005666861db5e178f56a2477d285aa/lib/python3.9/site-packages/wandb/sdk/wandb_setup.py", line 312, in s
etup
    ret = _setup(settings=settings)
  File "/home/user/.cache/pants/named_caches/pex_root/venvs/e33dbad2e381237c2ef87dd830f8fbcc73be777f/7898379b30005666861db5e178f56a2477d285aa/lib/python3.9/site-packages/wandb/sdk/wandb_setup.py", line 307, in _
setup
    wl = _WandbSetup(settings=settings)
  File "/home/user/.cache/pants/named_caches/pex_root/venvs/e33dbad2e381237c2ef87dd830f8fbcc73be777f/7898379b30005666861db5e178f56a2477d285aa/lib/python3.9/site-packages/wandb/sdk/wandb_setup.py", line 293, in _
_init__
    _WandbSetup._instance = _WandbSetup__WandbSetup(settings=settings, pid=pid)
  File "/home/user/.cache/pants/named_caches/pex_root/venvs/e33dbad2e381237c2ef87dd830f8fbcc73be777f/7898379b30005666861db5e178f56a2477d285aa/lib/python3.9/site-packages/wandb/sdk/wandb_setup.py", line 106, in _
_init__
    self._setup()
  File "/home/user/.cache/pants/named_caches/pex_root/venvs/e33dbad2e381237c2ef87dd830f8fbcc73be777f/7898379b30005666861db5e178f56a2477d285aa/lib/python3.9/site-packages/wandb/sdk/wandb_setup.py", line 234, in _
setup
    self._setup_manager()
  File "/home/user/.cache/pants/named_caches/pex_root/venvs/e33dbad2e381237c2ef87dd830f8fbcc73be777f/7898379b30005666861db5e178f56a2477d285aa/lib/python3.9/site-packages/wandb/sdk/wandb_setup.py", line 265, in _
setup_manager
    self._manager = wandb_manager._Manager(
  File "/home/user/.cache/pants/named_caches/pex_root/venvs/e33dbad2e381237c2ef87dd830f8fbcc73be777f/7898379b30005666861db5e178f56a2477d285aa/lib/python3.9/site-packages/wandb/sdk/wandb_manager.py", line 111, in
 __init__
    self._service.start()
  File "/home/user/.cache/pants/named_caches/pex_root/venvs/e33dbad2e381237c2ef87dd830f8fbcc73be777f/7898379b30005666861db5e178f56a2477d285aa/lib/python3.9/site-packages/wandb/sdk/service/service.py", line 119,
in start
    self._launch_server()
  File "/home/user/.cache/pants/named_caches/pex_root/venvs/e33dbad2e381237c2ef87dd830f8fbcc73be777f/7898379b30005666861db5e178f56a2477d285aa/lib/python3.9/site-packages/wandb/sdk/service/service.py", line 109,
in _launch_server
    internal_proc = subprocess.Popen(
  File "/home/user/Software/mambaforge/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/home/user/Software/mambaforge/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: '/tmp/pants-sandbox-6kYP6O/src.pex'
wandb: ERROR Abnormal program exit
```


### Additional Files

_No response_

### Environment

WandB version: 0.13.6

OS: Linux

Python version: 3.9

Versions of relevant libraries:


### Additional Context

_No response_


Discussion:
Hi @nharada1, thank you for writing in. We have built in support with launching wandb from within a [PEX environment](https://github.com/wandb/wandb/pull/4440) as of our latest client version. I will review your test case, reproduce the error, and report back. In the meantime could you try disabling wandb service,  `export WANDB_DISABLE_SERVICE=True` and update me on new behaviors/errors with your experiments. Thanks
Ahh, yes this worked with the service disabled, awesome! Thank you very much, do you know which release the Pex support will land in?

Resolving patch:
diff --git a/wandb/env.py b/wandb/env.py
--- a/wandb/env.py
+++ b/wandb/env.py
@@ -80,6 +80,7 @@
 INIT_TIMEOUT = "WANDB_INIT_TIMEOUT"
 GIT_COMMIT = "WANDB_GIT_COMMIT"
 GIT_REMOTE_URL = "WANDB_GIT_REMOTE_URL"
+_EXECUTABLE = "WANDB_EXECUTABLE"
 
 # For testing, to be removed in future version
 USE_V1_ARTIFACTS = "_WANDB_USE_V1_ARTIFACTS"
diff --git a/wandb/sdk/wandb_settings.py b/wandb/sdk/wandb_settings.py
--- a/wandb/sdk/wandb_settings.py
+++ b/wandb/sdk/wandb_settings.py
@@ -6,6 +6,7 @@
 import os
 import platform
 import re
+import shutil
 import socket
 import sys
 import tempfile
@@ -1396,9 +1397,11 @@ def _infer_settings_from_environment(
                 # chroot jails or docker containers. Return user id in these cases.
                 settings["username"] = str(os.getuid())
 
-        # one special case here is running inside a PEX environment,
-        # see https://pex.readthedocs.io/en/latest/index.html for more info about PEX
-        _executable = self._executable or os.environ.get("PEX") or sys.executable
+        _executable = (
+            os.environ.get(wandb.env._EXECUTABLE, self._executable)
+            or sys.executable
+            or shutil.which("python")
+        )
         if _executable is None or _executable == "":
             _executable = "python3"
         settings["_executable"] = _executable


Test patch: