Repository: spyder-ide/spyder

Issue:
Spyder can't start pylsp on pip installations
## Problem Description

Spyder doesn't manage to start pylsp, even though it's installed.

in `server_python_$PID.log`: <br/>
`C:\Program Files\Python310\pythonw.exe: No module named pylsp`

(there's also a corresponding GUI message after 5 or so start attempts)

however, pylsp seems to be installed correctly in a python interpreter, I can `import pylsp`, plus when I copy'n'paste the command that should start the python language server, that works. 

Therefore I suspect that the PYTHONPATH is set improperly inside spyder when attempting to start pylsp, without the pylsp directory <br/>
This is an installation on windows, for that matter. Pythonpath is:
```
sys.path
Out[2]: 
['C:\\Program Files\\Python310\\python310.zip',
 'C:\\Program Files\\Python310\\DLLs',
 'C:\\Program Files\\Python310\\lib',
 'C:\\Program Files\\Python310',
 '',
'%APPDATA%\\Python\\Python310\\site-packages',
'%APPDATA%\\Python\\Python310\\site-packages\\win32',
'%APPDATA%\\Python\\Python310\\site-packages\\win32\\lib',
'%APPDATA%\\Python\\Python310\\site-packages\\Pythonwin',
 'C:\\Program Files\\Python310\\lib\\site-packages',
'%APPDATA%\\Python\\Python310\\site-packages\\IPython\\extensions',
 '%USERPROFILE'%\\.ipython']
```
and pylsp is installed in `%APPDATA%\\Python\\Python310\\site-packages`
### What steps reproduce the problem?

1.  install python3.10.4
2.  pip install spyder
3.  launch spyder (in debug mode to get the error log, but happens without debug mode as well)

## Versions
<!--- You can get this information from Help > About Spyder...
or (if Spyder won't launch) the "conda list" command
from the Anaconda Prompt/Terminal/command line. --->

* Spyder version: 5.3.0 None
* Python version: 3.10.4 64-bit
* Qt version: 5.15.2
* PyQt5 version: 5.15.6
* Operating System: Windows 10

### Dependencies
<!--- Please go to the menu entry Help > Dependencies,
press the Copy to clipboard button and paste below --->

```
# Mandatory:
atomicwrites >=1.2.0          :  1.4.0 (OK)
chardet >=2.0.0               :  4.0.0 (OK)
cloudpickle >=0.5.0           :  2.0.0 (OK)
cookiecutter >=1.6.0          :  1.7.3 (OK)
diff_match_patch >=20181111   :  20200713 (OK)
intervaltree >=3.0.2          :  3.1.0 (OK)
IPython >=7.31.1;<8.0.0       :  7.32.0 (OK)
jedi >=0.17.2;<0.19.0         :  0.18.1 (OK)
jellyfish >=0.7               :  0.9.0 (OK)
jsonschema >=3.2.0            :  4.4.0 (OK)
keyring >=17.0.0              :  23.5.0 (OK)
nbconvert >=4.0               :  6.5.0 (OK)
numpydoc >=0.6.0              :  1.2.1 (OK)
paramiko >=2.4.0              :  2.10.3 (OK)
parso >=0.7.0;<0.9.0          :  0.8.3 (OK)
pexpect >=4.4.0               :  4.8.0 (OK)
pickleshare >=0.4             :  0.7.5 (OK)
psutil >=5.3                  :  5.9.0 (OK)
pygments >=2.0                :  2.11.2 (OK)
pylint >=2.5.0                :  2.13.5 (OK)
pyls_spyder >=0.4.0           :  0.4.0 (OK)
pylsp >=1.4.1;<1.5.0          :  1.4.1 (OK)
pylsp_black >=1.2.0           :  1.2.0 (OK)
qdarkstyle >=3.0.2;<3.1.0     :  3.0.3 (OK)
qstylizer >=0.1.10            :  0.2.1 (OK)
qtawesome >=1.0.2             :  1.1.1 (OK)
qtconsole >=5.3.0;<5.4.0      :  5.3.0 (OK)
qtpy >=2.0.1                  :  2.0.1 (OK)
rtree >=0.9.7                 :  1.0.0 (OK)
setuptools >=49.6.0           :  58.1.0 (OK)
sphinx >=0.6.6                :  4.5.0 (OK)
spyder_kernels >=2.3.0;<2.4.0 :  2.3.0 (OK)
textdistance >=4.2.0          :  4.2.2 (OK)
three_merge >=0.1.1           :  0.1.1 (OK)
watchdog >=0.10.3             :  2.1.7 (OK)
zmq >=17                      :  22.3.0 (OK)

# Optional:
cython >=0.21                 :  None (NOK)
matplotlib >=3.0.0            :  3.5.1 (OK)
numpy >=1.7                   :  1.22.3 (OK)
pandas >=1.1.1                :  1.4.2 (OK)
scipy >=0.17.0                :  1.8.0 (OK)
sympy >=0.7.3                 :  None (NOK)

```



Discussion:
Hi @arian-f thank you for the feedback! Could you reset you preferences running with the reset flag (`spyder --reset`)? Let us know if that helps
still reproduce it with a new spyder config directory (%userprofile%\.spyder-py3) - that should be equivalent, right?
Just to be sure maybe try then to run Spyder using the safe-mode flag (running with that will use a clean config directory without removing yours). For that you can run `spyder --safe-mode`. Let us know!
`spyder --safe-mode` shows the same thing.
Did you install Python with admin privileges? Could you start Spyder from a cmd with admin privileges and check if the pylsp starts?
I installed python itself as admin, but packages as unprivileged user -> packages that are not part of python itself are in `%appdata%\Python\Python310\site-packages`. When I run it with admin priviliges, it's still reproducing the error. However, I don't see how that would make a difference: it would be able to modify system files, but it should be able to read everything anyway.

However I have the feeling it would be more productive to inspect why it can't find the installed package instead of general debugging attempts. Easiest would be to somehow gain a python shell instead of the pylsp start. Could you give me a pointer on how to achieve that?

This is my work laptop -> I'm going off for easter now. Might look into this specifically over the weekend, but don't be surprised if I only manage to get back on Tuesday.

Well since seems like you are able to launch manually the pylsp you could try to use the Advanced configuration (`Tools > Preferences > Completion and linting > Advanced`) to see if Spyder is able to connect to the manually started pylsp server instance.

Since you have your interpreter and packages in different places most probably as you say is something regarding the sys.path. Checking a little bit I think Spyder handles that here: https://github.com/spyder-ide/spyder/blob/270f141944aadfbb623aa9208bd4b01a950c03f3/spyder/plugins/completion/providers/languageserver/client.py#L266-L319

Maybe you can check if unindenting there the line 308 (`logger.info('Server process env variables: {0}'.format(env.keys()))`) and running again in debug mode helps us get more info regarding how your env variables look for the pylsp process being launched.



@arian-f, are you setting your PYTHONPATH manually outside Spyder? Because if you do, then that should be the cause of this problem.
@ccordoba12 PYTHONPATH is not set.

The problem might be in python, however it can be solved in spyder: `start_server` starts a process with a clean environment, whereas `start_transport` does not. The clean environment causes python not to add `site.USER_SITE` to `sys.path` even though `site.ENABLE_USER_SITE` is True. From [site's documentation](https://docs.python.org/3/library/site.html#site.USER_SITE) on windows, `site.USER_SITE` is `%APPDATA%\Python\PythonXY\site-package` - this hints that `%APPDATA%` must be set. 

It's definitely confusing that `site.USER_SITE` points to the correct directory even if it's not added. That's why I think that the bug might be in python itself.

Indeed, adding `%APPDATA%` to the server environment fixes things.
Thus the fix in spyder is literally one line. I'd add it with a pull request, which branch should I base it on? `5.x`? 

The same issue might occur in unix and macos. I could try to check it on linux, but I don't have access to macos.


## details

I started a python shell instead of the server via:

```
$ diff -u Python39/site-packages/spyder/plugins/completion/providers/languageserver/client.py Python310/site-packages/spyder/plugins/completion/providers/languageserver/client.py
--- Python39/site-packages/spyder/plugins/completion/providers/languageserver/client.py 2022-04-19 10:16:16.536655900 +0200
+++ Python310/site-packages/spyder/plugins/completion/providers/languageserver/client.py        2022-04-19 12:07:10.120589200 +0200
@@ -316,7 +316,11 @@
             self.server.setStandardOutputFile(self.server_log_file)

         # Start server
-        self.server.start(self.server_args[0], self.server_args[1:])
+        # self.server.start(self.server_args[0], self.server_args[1:])
+        import win32api
+        exe = win32api.GetShortPathName(self.transport_args[0])
+        self.server.start('c:/windows/system32/cmd.exe', ['/c', f'start {exe}'])
```

and indead, sys.path doesn't have the `site.USER_SITE`: (sorry for the 8.3 names here, I didn't manage to start a program with spaces)
```>>> for k, v in os.environ.items(): print(f'{k}: {v}')
...
COMSPEC: c:\windows\system32\cmd.exe
PATH: %APPDATA%\Python\Python310\site-packages\PyQt5\Qt5\bin;%APPDATA%\Python\Python310\site-packages\PyQt5\Qt5\bin;%APPDATA%\Python\Python310\site-packages\PyQt5\Qt5\bin;%APPDATA%\Python\Python310\site-packages\PyQt5\Qt5\bin;%APPDATA%\Python\Python310\site-packages\PyQt5\Qt5\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\libnvvp;C:\Program Files\Python39\Scripts\;C:\Program Files\Python39\;C:\Program Files\Python310\Scripts\;C:\Program Files\Python310\;C:\Program Files\ImageMagick-7.1.0-Q16;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\PuTTY\;C:\Program Files\Git\cmd;C:\Program Files\dotnet\;C:\Program Files\NVIDIA Corporation\Nsight Compute 2022.1.1\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Users\asanusi.FLISOMAG\AppData\Local\Microsoft\WindowsApps;C:\Users\asanusi.FLISOMAG\.dotnet\tools;C:\Program Files\CMake\bin;C:\Program Files (x86)\Nmap
PATHEXT: .COM;.EXE;.BAT;.CMD;.VBS;.JS;.WS;.MSC
PROMPT: $P$G
SYSTEMROOT: C:\WINDOWS
USERPROFILE: C:\Users\User

>>> for p in sys.path: print(p)
C:\PROGRA~1\PYTHON~1\python310.zip
C:\PROGRA~1\PYTHON~1\DLLs
C:\PROGRA~1\PYTHON~1\lib
C:\PROGRA~1\PYTHON~1
C:\PROGRA~1\PYTHON~1\lib\site-packages

>>> sys.executable
'C:\\PROGRA~1\\PYTHON~1\\python.exe'

>>> site.USER_SITE
'%APPDATA%\\Python\\Python310\\site-packages'
```
Ok, thanks for investigating this further @arian-f! So, this problem would be solved if we add `APPDATA` to the env vars of the process that starts the server?
exactly, e.g. this diff makes it work:

```diff
$ diff -u Python39/site-packages/spyder/plugins/completion/providers/languageserver/client.py Python310/site-packages/spyder/plugins/completion/providers/languageserver/client.py
--- Python39/site-packages/spyder/plugins/completion/providers/languageserver/client.py 2022-04-19 12:33:22.461020400 +0200
+++ Python310/site-packages/spyder/plugins/completion/providers/languageserver/client.py        2022-04-19 13:12:20.711573600 +0200
@@ -297,6 +297,7 @@
             # the PyLS.
             if os.name == "nt" and "USERPROFILE" in os.environ:
                 env.insert("USERPROFILE", os.environ["USERPROFILE"])
+            if os.name == "nt" and "APPDATA" in os.environ:
+                env.insert("APPDATA", os.environ["APPDATA"])
         else:
             # There's no need to define a cwd for other servers.
             cwd = None
```
Ok, thanks for the confirmation @arian-f! I'm going to do this only for non-Anaconda installations because packages in `USER_SITE` pollute (and usually break) all conda environments.

That's why we prefer to start the server with the minimal amount of env vars, i.e. so that it can't be broken by packages installed outside the environment in which it's present.

Resolving patch:
diff --git a/spyder/plugins/completion/providers/languageserver/client.py b/spyder/plugins/completion/providers/languageserver/client.py
--- a/spyder/plugins/completion/providers/languageserver/client.py
+++ b/spyder/plugins/completion/providers/languageserver/client.py
@@ -28,7 +28,9 @@
 # Local imports
 from spyder.api.config.mixins import SpyderConfigurationAccessor
 from spyder.config.base import (
-    DEV, get_conf_path, get_debug_level, running_in_ci, running_under_pytest)
+    DEV, get_conf_path, get_debug_level, is_pynsist, running_in_ci,
+    running_under_pytest)
+from spyder.config.utils import is_anaconda
 from spyder.plugins.completion.api import (
     CLIENT_CAPABILITES, SERVER_CAPABILITES,
     TEXT_DOCUMENT_SYNC_OPTIONS, CompletionRequestTypes,
@@ -291,12 +293,23 @@ def start_server(self):
             if not osp.exists(cwd):
                 os.makedirs(cwd)
 
-            # On Windows, some modules (notably Matplotlib)
-            # cause exceptions if they cannot get the user home.
-            # So, we need to pass the USERPROFILE env variable to
-            # the PyLS.
-            if os.name == "nt" and "USERPROFILE" in os.environ:
-                env.insert("USERPROFILE", os.environ["USERPROFILE"])
+            if os.name == "nt":
+                # On Windows, some modules (notably Matplotlib)
+                # cause exceptions if they cannot get the user home.
+                # So, we need to pass the USERPROFILE env variable to
+                # the PyLSP.
+                if "USERPROFILE" in os.environ:
+                    env.insert("USERPROFILE", os.environ["USERPROFILE"])
+
+                # The PyLSP can't start on pip installations if APPDATA
+                # is missing and the user has installed their packages on
+                # that directory.
+                # Fixes spyder-ide/spyder#17661
+                if (
+                    not (is_anaconda() or is_pynsist())
+                    and "APPDATA" in os.environ
+                ):
+                    env.insert("APPDATA", os.environ["APPDATA"])
         else:
             # There's no need to define a cwd for other servers.
             cwd = None


Test patch: