Repository: ipython/ipython

Issue:
Win32 shell interactivity apparently broke qtconsole "cd" magic
PR #1424, Win32 shell interactivity, was accepted as not enhancing qtconsole, but was not thought to break it. It seems to have. This was reproduced on several different systems.  Located the breakage by bisection at 4953f2d

@mwiebe, @minrk could you have a look?

On a Win7-64 native box, with no recent history of running "ipython kernel", the following commands immediately after qtconsole startup

```
ls
cd ..
```

reliably caused the following traceback, followed by general odd behavior, apparent corruption, until restart qtconsole.

```
TypeError                                 Traceback (most recent call last)
c:\Users\jmarch\<ipython-input-2-56af63bcfcf6> in <module>()
----> 1 get_ipython().magic(u'cd ..')

c:\users\jmarch\ipython\IPython\core\interactiveshell.pyc in magic(self, arg_s, next_input)
   2034                 self._magic_locals = sys._getframe(1).f_locals
   2035             with self.builtin_trap:
-> 2036                 result = fn(magic_args)
   2037             # Ensure we're not keeping object references around:
   2038             self._magic_locals = {}

c:\users\jmarch\ipython\IPython\core\magic.pyc in magic_cd(self, parameter_s)
   2953             # for c:\windows\directory\names\
   2954             parameter_s = re.sub(r'\\(?! )','/', parameter_s)
-> 2955             opts,ps = self.parse_options(parameter_s,'qb',mode='string')
   2956         # jump to previous
   2957         if ps == '-':

c:\users\jmarch\ipython\IPython\core\magic.pyc in parse_options(self, arg_str, opt_str, *long_opts, **kw)
    279             # If the list of inputs only has 0 or 1 thing in it, there's no
    280             # need to look for options
--> 281             argv = arg_split(arg_str, posix, strict)
    282             # Do regular option processing
    283             try:

c:\users\jmarch\ipython\IPython\utils\_process_win32.pyc in arg_split(commandline, posix, strict)
    178         result_pointer = CommandLineToArgvW(py3compat.cast_unicode(commandline.lstrip()), ctypes.byref(argvn))
    179         result_array_type = LPCWSTR * argvn.value
--> 180         result = [arg for arg in result_array_type.from_address(result_pointer)]
    181         retval = LocalFree(result_pointer)
    182         return result

TypeError: integer expected
```

After running "ipython kernel" to test this on a 2-process terminal, I was unable to again reproduce the problem on this system even in a simple qtconsole. Have not yet rebooted.

Easiest short-term workaround may be to revert to previous behavior in _process_win32.system when running qtconsole?



Discussion:
I can't test this right now, but I'm raising it to blocker for 0.13, b/c we can't really release with the kind of breakage described here...

FWIW, the same error appears in a test result on Windows: https://jenkins.shiningpanda.com/ipython/job/ipython-windows-py27/lastCompletedBuild/testReport/IPython.utils.tests/test_process/test_arg_split_win32/

I think the arg_split code that calls ctypes was @jstenar's, so he might have some idea why it's failing.

Update: breakage reappears after reboot, and continues after a small amount of 2-process terminal activity.

I tested this on the build I had, and couldn't reproduce it. Updating to the latest master gives me that exact error, though. My previous build had the shell interactivity patch in it, so I would have expected to have the error already without updating.

The same commit (4953f2d) also broke the "ls" command from the frontend of a 2-process terminal session in Windows

```
$ ipython kernel
...
$ ipython console --existing
> ls
```

After the frontend prints the listing, it does not return to a prompt, and becomes unresponsive except to console-killing Ctrl+C. (The kernel seems to freeze).

I can reproduce this as well, _only_ when the kernel is started with `ipython kernel`.  I cannot reproduce it with regular `ipython [qt]console`, or even using `--existing` to connect to another kernel, if that kernel was started with an attached frontend.

I haven't the foggiest idea how or why this could be.

If we don't figure out a real solution by the time we want to release, then we can certainly just use the old version in the kernel as a band-aid.

Since the error is appearing in code that's used to parse a command line, I wonder if the process's own command line could be interfering with it. Then any difference between manually starting `ipython kernel` and the mechanism the Qt console uses to start it in a subprocess could come into play.

Good point, thanks.

Possibly related, I see the following test now hanging on Windows:

`IPython.utils.tests.test_process.SubProcessTestCase.test_system`

Here's a minimal patch to revert the functionality. I suspect the issues have to do with the fact that there isn't a good cross-platform way to asynchronously handle stdin/stdout. This particular implementation was working ok in all the testing I did, but I guess there's more depth to the issue that needs to be explored.

```
diff --git a/IPython/utils/_process_win32.py b/IPython/utils/_process_win32.py
index 7ee9b62..85ee0d0 100644
--- a/IPython/utils/_process_win32.py
+++ b/IPython/utils/_process_win32.py
@@ -124,8 +124,13 @@ def system(cmd):
     """
     # The controller provides interactivity with both
     # stdin and stdout
-    import _process_win32_controller
-    _process_win32_controller.system(cmd)
+    #import _process_win32_controller
+    #_process_win32_controller.system(cmd)
+
+    with AvoidUNCPath() as path:
+        if path is not None:
+            cmd = '"pushd %s &&"%s' % (path, cmd)
+        return process_handler(cmd, _system_body)
```


Resolving patch:
diff --git a/IPython/utils/_process_win32.py b/IPython/utils/_process_win32.py
--- a/IPython/utils/_process_win32.py
+++ b/IPython/utils/_process_win32.py
@@ -125,9 +125,13 @@ def system(cmd):
     """
     # The controller provides interactivity with both
     # stdin and stdout
-    import _process_win32_controller
-    _process_win32_controller.system(cmd)
+    #import _process_win32_controller
+    #_process_win32_controller.system(cmd)
 
+    with AvoidUNCPath() as path:
+        if path is not None:
+            cmd = '"pushd %s &&"%s' % (path, cmd)
+        return process_handler(cmd, _system_body)
 
 def getoutput(cmd):
     """Return standard output of executing cmd in a shell.


Test patch: