Repository: xonsh/xonsh

Issue:
pytest plugin causes issues on Windows Azure Pipelines
```
py38 inst-nodeps: D:\a\1\s\.tox\.tmp\package\1\virtualenv-20.0.17.dev4+g3a441c8.tar.gz
py38 installed: appdirs==1.4.3,atomicwrites==1.3.0,attrs==19.3.0,colorama==0.4.3,coverage==5.0.4,distlib==0.3.0,filelock==3.0.12,more-itertools==8.2.0,packaging==20.3,pluggy==0.13.1,py==1.8.1,pyparsing==2.4.7,pytest==5.4.1,pytest-env==0.6.2,pytest-mock==2.0.0,pytest-timeout==1.3.4,six==1.14.0,virtualenv==20.0.17.dev4+g3a441c8,wcwidth==0.1.9,xonsh==0.9.15
py38 run-test-pre: PYTHONHASHSEED='343'
py38 run-test: commands[0] | python -c 'from os.path import sep; file = open(r"D:\a\1\s\.tox\py38\Lib\site-packages{}coverage-virtualenv.pth".format(sep), "w"); file.write("import coverage; coverage.process_startup()"); file.close()'
py38 run-test: commands[1] | python -m coverage erase
py38 run-test: commands[2] | python -m coverage run -m pytest --junitxml 'D:\a\1\s\.tox/junit.py38.xml' --timeout 30 tests --int
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "D:\a\1\s\.tox\py38\lib\site-packages\_pytest\main.py", line 187, in wrap_session
INTERNALERROR>     config._do_configure()
INTERNALERROR>   File "D:\a\1\s\.tox\py38\lib\site-packages\_pytest\config\__init__.py", line 820, in _do_configure
INTERNALERROR>     self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR>   File "D:\a\1\s\.tox\py38\lib\site-packages\pluggy\hooks.py", line 308, in call_historic
INTERNALERROR>     res = self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "D:\a\1\s\.tox\py38\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "D:\a\1\s\.tox\py38\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "D:\a\1\s\.tox\py38\lib\site-packages\pluggy\callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "D:\a\1\s\.tox\py38\lib\site-packages\pluggy\callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "D:\a\1\s\.tox\py38\lib\site-packages\pluggy\callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "D:\a\1\s\.tox\py38\lib\site-packages\xonsh\pytest_plugin.py", line 13, in pytest_configure
INTERNALERROR>     setup()
INTERNALERROR>   File "D:\a\1\s\.tox\py38\lib\site-packages\xonsh\__amalgam__.py", line 24306, in setup
INTERNALERROR>     execer = Execer(xonsh_ctx=ctx)
INTERNALERROR>   File "D:\a\1\s\.tox\py38\lib\site-packages\xonsh\__amalgam__.py", line 23191, in __init__
INTERNALERROR>     load_builtins(execer=self, ctx=xonsh_ctx)
INTERNALERROR>   File "D:\a\1\s\.tox\py38\lib\site-packages\xonsh\__amalgam__.py", line 22923, in load_builtins
INTERNALERROR>     builtins.__xonsh__.link_builtins(execer=execer)
INTERNALERROR>   File "D:\a\1\s\.tox\py38\lib\site-packages\xonsh\__amalgam__.py", line 23054, in link_builtins
INTERNALERROR>     builtins.default_aliases = builtins.aliases = Aliases(make_default_aliases())
INTERNALERROR>   File "D:\a\1\s\.tox\py38\lib\site-packages\xonsh\__amalgam__.py", line 21585, in make_default_aliases
INTERNALERROR>     if not locate_binary("sudo"):
INTERNALERROR>   File "D:\a\1\s\.tox\py38\lib\site-packages\xonsh\__amalgam__.py", line 19615, in locate_binary
INTERNALERROR>     return builtins.__xonsh__.commands_cache.locate_binary(name)
INTERNALERROR>   File "D:\a\1\s\.tox\py38\lib\site-packages\xonsh\__amalgam__.py", line 8780, in locate_binary
INTERNALERROR>     _ = self.all_commands
INTERNALERROR>   File "D:\a\1\s\.tox\py38\lib\site-packages\xonsh\__amalgam__.py", line 8723, in all_commands
INTERNALERROR>     for cmd in executables_in(path):
INTERNALERROR>   File "D:\a\1\s\.tox\py38\lib\site-packages\xonsh\__amalgam__.py", line 5226, in executables_in
INTERNALERROR>     yield from func(path)
INTERNALERROR>   File "D:\a\1\s\.tox\py38\lib\site-packages\xonsh\__amalgam__.py", line 5205, in _executables_in_windows
INTERNALERROR>     for x in scandir(path):
INTERNALERROR> FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Program Files '
ERROR: InvocationError for command 'D:\a\1\s\.tox\py38\Scripts\python.EXE' -m coverage run -m pytest --junitxml 'D:\a\1\s\.tox/junit.py38.xml' --timeout 30 tests --int (exited with code 3)
```
@scopatz this is sadly causing issues in the virtualenv package.


Discussion:
It looks like the Azure Agent images has "C:\Program Files " (note the trailing space) in the PATH. 

This is probably just a typo, and note something that Windows seem to care about. It seems paths with spaces at the end are just stripped. 

Unfortunately, python is not consistent about this. The `os.path.isdir()` check we do in [here](https://github.com/xonsh/xonsh/blob/0.9.15/xonsh/tools.py#L792) seems to think that the path is valid. But `scandir()` a [few lines later](https://github.com/xonsh/xonsh/blob/0.9.15/xonsh/tools.py#L803) is not so forgiving. 

![image](https://user-images.githubusercontent.com/1038978/78644904-f0f21780-78b6-11ea-8d85-e2a02a930a88.png)






Can we handle the exception? I don't think a bad path should blow up like this 🤔 
I made a PR with a fix to handle this. Thanks a lot for reporting this @gaborbernat. It really helps a lot to find these edge cases. 
@melund that's great news. Can we merge this and do a release soon? It's blocking at the moment virtualenv CI... and I'd rather not drop testing xonsh compatibility to fix the CI... I've reported the issue to Azure... but that will take time. But I intend to fix the CI somehow very soon.
We'll need to reopen this, post-release the issue still manifest:

```
INTERNALERROR> FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Program Files '
```
The (partially) good news is that the issue now manifests on the CI here too, so maybe easier to validate the fix.
Notified Azure via https://twitter.com/AzureSupport/status/1247493592475734017 (but that will take a while to fix, and we should probably handle better this error).
The question now is: Is this a bug at all? Turns out that Windows do care about the trailing space after all...
See, for instance, [this question](https://superuser.com/questions/565334/rename-delete-windows-x64-folder-with-leading-and-trailing-space) at StackExchange.
I don't think we should strip spaces in entries on the ``PATH``, but not fail if some path entries there do not exist. 
Sigh :roll_eyes:
Fun fact:
```
>>> os.path.isdir("C:\Program Files ")
True
>>> os.path.normpath("C:\Program Files ")
'C:\\Program Files '
>>> os.path.realpath("C:\Program Files ")
'C:\\Program Files'
```
It is the very `os.stat()` that treats the path with trailing spaces as the one without them.
```
>>> os.stat("C:\Program Files     ")
os.stat_result(st_mode=16749, st_ino=281474976710717, st_dev=173025906, st_nlink
=1, st_uid=0, st_gid=0, st_size=8192, st_atime=1586154685, st_mtime=1586154685,
st_ctime=1377178576)
```
I don't really know what we could do about it except for reimplementing the stat call on Windows.
Perhaps we could take a deep breath, close our eyes and just do
```python
try:
    for x in scandir(path):
        ...
except FileNotFoundError:
    return
```

Resolving patch:
diff --git a/xonsh/tools.py b/xonsh/tools.py
--- a/xonsh/tools.py
+++ b/xonsh/tools.py
@@ -791,9 +791,6 @@ def _executables_in_posix(path):
 def _executables_in_windows(path):
     if not os.path.isdir(path):
         return
-    # isdir will return True even with a trailing space
-    # scandir doesn't so normalize path first
-    path = os.path.normpath(path)
     extensions = builtins.__xonsh__.env["PATHEXT"]
     if PYTHON_VERSION_INFO < (3, 5, 0):
         for fname in os.listdir(path):
@@ -803,18 +800,24 @@ def _executables_in_windows(path):
                 if ext.upper() in extensions:
                     yield fname
     else:
-        for x in scandir(path):
-            try:
-                is_file = x.is_file()
-            except OSError:
-                continue
-            if is_file:
-                fname = x.name
-            else:
-                continue
-            base_name, ext = os.path.splitext(fname)
-            if ext.upper() in extensions:
-                yield fname
+        try:
+            for x in scandir(path):
+                try:
+                    is_file = x.is_file()
+                except OSError:
+                    continue
+                if is_file:
+                    fname = x.name
+                else:
+                    continue
+                base_name, ext = os.path.splitext(fname)
+                if ext.upper() in extensions:
+                    yield fname
+        except FileNotFoundError:
+            # On Windows, there's no guarantee for the directory to really
+            # exist even if isdir returns True. This may happen for instance
+            # if the path contains trailing spaces.
+            return
 
 
 def executables_in(path):


Test patch: