Repository: spyder-ide/spyder

Issue:
Can't run files with space in path via external terminal
I dumped the cmd being called from spyder.util.programs (line 764ish) with a simple print statement, and nothing seems immediately wrong, but `[WinError 267] The directory name is invalid` is consistently thrown.

Cmd:
> start cmd.exe /K "cd "Y:\Users\aaron\python scripts" && "C:\Users\aaron\python\py39\python.exe" -i "Y:\Users\aaron\python scripts\helloworld.py"" ^&^& exit

I can even separately copy and paste the command into a different terminal (just python inside cmd; no IDE) and call the following without issue:

```
from subprocess import Popen
Popen(r'start cmd.exe /K "cd "Y:\Users\aaron\python scripts" && "C:\Users\aaron\python\py39\python.exe" -i "Y:\Users\aaron\python scripts\helloworld.py"" ^&^& exit', shell=True)
```
Including the `cwd` arg does not make a difference either when using Popen myself, but if I modify spyder.utils.programs.py at line 762 to be:
```
        try:
            # if wdir:
            #     run_shell_command(cmd, cwd=wdir)
            # else:
                run_shell_command(cmd)
```
the problem is solved.

Am I missing something, or does the "cd" command included in the `cmd` string take care of not needing the `cwd` argument? I also cannot figure out why it's an issue for spyder to include `cwd` but not when I do it in a terminal myself...

* Spyder version: 5.3.0 None
* Python version: 3.9.6 64-bit
* Qt version: 5.12.10
* PyQt5 version: 5.12.3
* Operating System: Windows 10


Discussion:
Hey @athompson673, thanks for reporting. You said

> does the "cd" command included in the cmd string take care of not needing the cwd argument?

It seems it does, so we don't need to pass it again to `run_shell_command`.

@dalthviz, could you test that on Windows to see if that's the case? Thanks!
This seems like it may also affect linux systems at [line 795](https://github.com/spyder-ide/spyder/blob/master/spyder/utils/programs.py#L795)? IDK, I don't have a linux PC handy.
> This seems like it may also affect linux systems at [line 795](https://github.com/spyder-ide/spyder/blob/master/spyder/utils/programs.py#L795)? IDK, I don't have a linux PC handy.

It does not. For Linux systems (lines 770-794), the script file does not include `cd`, but `run_program` does include `cwd`; here `wdir` is not a quoted string. For macOS (lines 795-815), the script does include `cd`, but the `run_shell_command` does not include `cwd` keyword argument.
@ccordoba12 I was able to reproduce this on Windows. If I run a file that has a space in his path I get the following message: 

![imagen](https://user-images.githubusercontent.com/16781833/161602559-4a02f4c0-7a53-44f5-aa4e-940a2fa4c211.png)

If the file path doesn't have a space it runs as expected

Will it work on windows if there are spaces in `wdir` but we don't quote it? i.e. `cwd='path with space'` instead of `cwd='"path with space"'`. It may be more pythonic to remove `cd` from temporary script and use the `cwd` keyword argument.
> here `wdir` is not a quoted string.

Isn't it? The quotes get added before the OS switching at [line 737](https://github.com/spyder-ide/spyder/blob/master/spyder/utils/programs.py#L737)



> Will it work on windows if there are spaces in `wdir` but we don't quote it? i.e. `cwd='path with space'`

I believe it is *required* to be un-quoted when it is passed as the `cwd` argument to Popen (inside `run_shell_command`) at least on Windows, however it seems like Linux and MacOS scripts may still need `wdir` to be quoted because they are used to build shell commands as strings / shell script. I don't have linux on any of my computers, but I think it needs to be tested.

The linux logic could possibly be simplified as it seems specifying the `"wdir-option"` in the command args as well as passing `wdir` to `run_shell_command` seems redundant like the Windows case.

> > here `wdir` is not a quoted string.
> 
> Isn't it? The quotes get added before the OS switching at [line 737](https://github.com/spyder-ide/spyder/blob/master/spyder/utils/programs.py#L737)

You are correct. I miss-read the code.
 
> > Will it work on windows if there are spaces in `wdir` but we don't quote it? i.e. `cwd='path with space'`
> 
> I believe it is _required_ to be un-quoted when it is passed as the `cwd` argument to Popen (inside `run_shell_command`) at least on Windows, however it seems like Linux and MacOS scripts may still need `wdir` to be quoted because they are used to build shell commands as strings / shell script. I don't have linux on any of my computers, but I think it needs to be tested.
> 
> The linux logic could possibly be simplified as it seems specifying the `"wdir-option"` in the command args as well as passing `wdir` to `run_shell_command` seems redundant like the Windows case.

Yes, I think it most pythonic to use the `cwd` keyword argument wherever possible; it likely requires unquoted string for all platforms, but should be verified. For linux, then, `wdir-option` could be removed and `cd` should be removed from both nt and darwin, with `cwd` added to the darwin case. The temporary file may not be necessary for darwin either, but I'd have to verify the interaction of Terminal.app with the provided `env` keyword for `Popen`...
@mrclary, could you give us a hand with this one, since it seems you understand well what needs to be done? Thanks!

Resolving patch:
diff --git a/spyder/utils/programs.py b/spyder/utils/programs.py
--- a/spyder/utils/programs.py
+++ b/spyder/utils/programs.py
@@ -32,7 +32,7 @@
 # Local imports
 from spyder.config.base import (running_under_pytest, get_home_dir,
                                 running_in_mac_app)
-from spyder.py3compat import PY2, is_text_string, to_text_string
+from spyder.py3compat import is_text_string, to_text_string
 from spyder.utils import encoding
 from spyder.utils.misc import get_python_executable
 
@@ -730,40 +730,23 @@ def run_python_script_in_terminal(fname, wdir, args, interact,
     if executable is None:
         executable = get_python_executable()
 
-    # If fname or wdir has spaces, must be enclosed in quotes (all platforms)
-    if ' ' in fname:
-        fname = '"' + fname + '"'
-    if ' ' in wdir:
-        wdir = '"' + wdir + '"'
+    # Quote fname in case it has spaces (all platforms)
+    fname = f'"{fname}"'
 
-    # If python_exe contains spaces, it can't be ran on Windows, so we
-    # have to enclose them in quotes. Also wdir can come with / as os.sep, so
-    # we need to take care of it.
-    if os.name == 'nt':
-        wdir = wdir.replace('/', '\\')
-        executable = '"' + executable + '"'
+    wdir = None if not wdir else wdir  # Cannot be empty string
 
-    p_args = [executable]
-    p_args += get_python_args(fname, python_args, interact, debug, args)
+    p_args = get_python_args(fname, python_args, interact, debug, args)
 
     if os.name == 'nt':
-        cmd = 'start cmd.exe /K "'
-        if wdir:
-            cmd += 'cd ' + wdir + ' && '
+        if wdir is not None:
+            # wdir can come with / as os.sep, so we need to take care of it.
+            wdir = wdir.replace('/', '\\')
+
+        # python_exe must be quoted in case it has spaces
+        cmd = f'start cmd.exe /K ""{executable}" '
         cmd += ' '.join(p_args) + '"' + ' ^&^& exit'
-        # Command line and cwd have to be converted to the filesystem
-        # encoding before passing them to subprocess, but only for
-        # Python 2.
-        # See https://bugs.python.org/issue1759845#msg74142 and
-        # spyder-ide/spyder#1856.
-        if PY2:
-            cmd = encoding.to_fs_from_unicode(cmd)
-            wdir = encoding.to_fs_from_unicode(wdir)
         try:
-            if wdir:
-                run_shell_command(cmd, cwd=wdir)
-            else:
-                run_shell_command(cmd)
+            run_shell_command(cmd, cwd=wdir)
         except WindowsError:
             from qtpy.QtWidgets import QMessageBox
             from spyder.config.base import _
@@ -772,44 +755,30 @@ def run_python_script_in_terminal(fname, wdir, args, interact,
                                    "an external terminal"),
                                  QMessageBox.Ok)
     elif sys.platform.startswith('linux'):
-        programs = [{'cmd': 'gnome-terminal',
-                     'wdir-option': '--working-directory',
-                     'execute-option': '-x'},
-                    {'cmd': 'konsole',
-                     'wdir-option': '--workdir',
-                     'execute-option': '-e'},
-                    {'cmd': 'xfce4-terminal',
-                     'wdir-option': '--working-directory',
-                     'execute-option': '-x'},
-                    {'cmd': 'xterm',
-                     'wdir-option': None,
-                     'execute-option': '-e'},]
+        programs = [{'cmd': 'gnome-terminal', 'execute-option': '-x'},
+                    {'cmd': 'konsole', 'execute-option': '-e'},
+                    {'cmd': 'xfce4-terminal', 'execute-option': '-x'},
+                    {'cmd': 'xterm', 'execute-option': '-e'}]
         for program in programs:
             if is_program_installed(program['cmd']):
-                arglist = []
-                if program['wdir-option'] and wdir:
-                    arglist += [program['wdir-option'], wdir]
-                arglist.append(program['execute-option'])
-                arglist += p_args
-                if wdir:
-                    run_program(program['cmd'], arglist, cwd=wdir)
-                else:
-                    run_program(program['cmd'], arglist)
+                cmd = [program['cmd'], program['execute-option'], executable]
+                cmd.extend(p_args)
+                run_shell_command(' '.join(cmd), cwd=wdir)
                 return
     elif sys.platform == 'darwin':
         f = tempfile.NamedTemporaryFile('wt', prefix='run_spyder_',
                                         suffix='.sh', dir=get_temp_dir(),
                                         delete=False)
         if wdir:
-            f.write('cd {}\n'.format(wdir))
+            f.write('cd "{}"\n'.format(wdir))
         if running_in_mac_app(executable):
-            f.write(f'export PYTHONHOME={os.environ["PYTHONPATH"]}\n')
-        f.write(' '.join(p_args))
+            f.write(f'export PYTHONHOME={os.environ["PYTHONHOME"]}\n')
+        f.write(' '.join([executable] + p_args))
         f.close()
         os.chmod(f.name, 0o777)
 
         def run_terminal_thread():
-            proc = run_shell_command('open -a Terminal.app ' + f.name, env={})
+            proc = run_shell_command(f'open -a Terminal.app {f.name}')
             # Prevent race condition
             time.sleep(3)
             proc.wait()


Test patch: