Repository: ipython/ipython

Issue:
Glob expansion tests fail on Windows
Both 2.7 and 3.2 are failing. See ShiningPanda output: https://jenkins.shiningpanda-ci.com/ipython/job/ipython-win-py32/30/console

I'm pretty sure this was caused by the recent merge of #2165.



Discussion:
Failure:

```
======================================================================
FAIL: Doctest: IPython.core.tests.test_run.doctest_run_option_parser
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\Users\slave\env\cpython-3.2\lib\site-packages\ipython-0.14.dev-py3.2.egg\IPython\testing\plugin\ipdoctest.py", line 271, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for IPython.core.tests.test_run.doctest_run_option_parser
  File "c:\Users\slave\env\cpython-3.2\lib\site-packages\ipython-0.14.dev-py3.2.egg\IPython\core\tests\test_run.py", line 90, in doctest_run_option_parser

----------------------------------------------------------------------
File "c:\Users\slave\env\cpython-3.2\lib\site-packages\ipython-0.14.dev-py3.2.egg\IPython\core\tests\test_run.py", line 99, in IPython.core.tests.test_run.doctest_run_option_parser
Failed example:
    get_ipython().magic('run print_argv.py print\\\\*.py')
Expected:
    ['print*.py']
Got:
    ['print\\*.py']
----------------------------------------------------------------------
File "c:\Users\slave\env\cpython-3.2\lib\site-packages\ipython-0.14.dev-py3.2.egg\IPython\core\tests\test_run.py", line 102, in IPython.core.tests.test_run.doctest_run_option_parser
Failed example:
    get_ipython().magic("run print_argv.py 'print*.py'")
Expected:
    ['print_argv.py']
Got:
    ["'print*.py'"]

>>  raise self.failureException(self.format_failure(<_io.StringIO object at 0x00000000038AE8B8>.getvalue()))
```

Test case:

``` python
def doctest_run_option_parser():
    r"""Test option parser in %run.

    In [1]: %run print_argv.py
    []

    In [2]: %run print_argv.py print*.py
    ['print_argv.py']

    In [3]: %run print_argv.py print\\*.py
    ['print*.py']

    In [4]: %run print_argv.py 'print*.py'
    ['print_argv.py']

    In [5]: %run -G print_argv.py print*.py
    ['print*.py']

    """
```

Ah, you can't escape glob by backslash in windows as it is path separator... I didn't think about that.  I think we should separate the doctest `In [3]` and wrap it with `@dec.skip_win32`.

I don't understand why the test `In [4]` fails.  Actually, it works in the way we wanted.  I don't have windows box so I can't debug this.  Does anybody know how glob works on windows?

Ah, I guess `In [4]` failure is due to the different option to `shlex.split`  So it should be wrapped by `@dec.skip_win32`, too.

There's another failure below:

```
======================================================================
FAIL: Test glob expansion for %run magic.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\Users\slave\env\cpython-3.2\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\Users\slave\env\cpython-3.2\lib\site-packages\ipython-0.14.dev-py3.2.egg\IPython\utils\tests\test_path.py", line 479, in test_shellglob
    assert_match([r'\*'], ['*'])
  File "c:\Users\slave\env\cpython-3.2\lib\site-packages\ipython-0.14.dev-py3.2.egg\IPython\utils\tests\test_path.py", line 468, in assert_match
    sorted(matches))
AssertionError: Lists differ: ['\\$Recycle.Bin', '\\Document... != ['*']

First differing element 0:
\$Recycle.Bin
*

First list contains 19 additional elements.
First extra element 1:
\Documents and Settings

+ ['*']
- ['\\$Recycle.Bin',
-  '\\Documents and Settings',
-  '\\MSOCache',
-  '\\PerfLogs',
-  '\\Program Files',
-  '\\Program Files (x86)',
-  '\\ProgramData',
-  '\\Recovery',
-  '\\System Volume Information',
-  '\\Users',
-  '\\Windows',
-  '\\jython2.5.1',
-  '\\pagefile.sys',
-  '\\python24',
-  '\\python25',
-  '\\python26',
-  '\\python27',
-  '\\python31',
-  '\\python32',
-  '\\sp']
    """Fail immediately, with the given message."""
```

Is there a way for a Windows user to escape `*` in %run arguments, if backslash doesn't work?

Oops I missed it.  `TemporaryDirectory` doesn't work as I expected here.  All I expect is that it returns chdir'able full path...  I really don't get it.

Windows user can use quote to skip escape as shlex won't strip it off.

```
In [3]: shlex.split("'a' 'b'", posix=False)
Out[3]: ["'a'", "'b'"]
```

(But we have another reasons for not using posix=False in posix systems:
http://docs.python.org/library/shlex.html#parsing-rules)

So in windows, quote acts more naturally:

``` python
def doctest_run_option_parser_for_windows():
    r"""Test option parser in %run.

    In [4]: %run print_argv.py 'print*.py'
    ['print_*.py']

    """
```

I'm out of town for a few days and won't be able to pitch in.  Feel free to revert or commit some `@skip`s if necessary in the mean time.

Can someone try #2478 on windows?

@tkf, I'm afraid I'm still seeing test failures from the test run this morning.

```
======================================================================
FAIL: IPython.utils.tests.test_path.TestShellGlob.test_match_windows(['a\\*', 'a*'], ['a\\*', 'a0', 'a1', 'a2'])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\Users\slave\Jenkins\shiningpanda\jobs\d5f643a2\virtualenvs\ff035a1d\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "S:\Users\slave\Jenkins\shiningpanda\jobs\d5f643a2\virtualenvs\ff035a1d\lib\site-packages\ipython-0.14.dev-py2.7.egg\IPython\utils\tests\test_path.py", line 484, in check_match
    sorted(matches))
AssertionError: Lists differ: ['a*', 'a0', 'a1', 'a2'] != ['a0', 'a1', 'a2', 'a\\*']

First differing element 0:
a*
a0

- ['a*', 'a0', 'a1', 'a2']
+ ['a0', 'a1', 'a2', 'a\\*']
    """Fail immediately, with the given message."""
>>  raise self.failureException("Lists differ: ['a*', 'a0', 'a1', 'a2'] != ['a0', 'a1', 'a2', 'a\\\\*']\n\nFirst differing element 0:\na*\na0\n\n- ['a*', 'a0', 'a1', 'a2']\n+ ['a0', 'a1', 'a2', 'a\\\\*']")


======================================================================
FAIL: IPython.utils.tests.test_path.TestShellGlob.test_match_windows(['a\\[012]'], ['a\\[012]'])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\Users\slave\Jenkins\shiningpanda\jobs\d5f643a2\virtualenvs\ff035a1d\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "S:\Users\slave\Jenkins\shiningpanda\jobs\d5f643a2\virtualenvs\ff035a1d\lib\site-packages\ipython-0.14.dev-py2.7.egg\IPython\utils\tests\test_path.py", line 484, in check_match
    sorted(matches))
AssertionError: Lists differ: ['a[012]'] != ['a\\[012]']

First differing element 0:
a[012]
a\[012]

- ['a[012]']
+ ['a\\[012]']
?    ++

    """Fail immediately, with the given message."""
>>  raise self.failureException("Lists differ: ['a[012]'] != ['a\\\\[012]']\n\nFirst differing element 0:\na[012]\na\\[012]\n\n- ['a[012]']\n+ ['a\\\\[012]']\n?    ++\n")
```

So, it seems there is a problem I was not aware of.  What `r'a\*'` should be interpreted in Windows?  Is it (1) "a" + "escaped star" or (2) "a" + "path separator" + "star"?  I am writing test for (2) but `shellglob` is doing (1).  I think we should choose (2) and change `shellglob` (i.e., do not unescape patterns when `sys.platform == 'win32'`), as (1) is impossible in windows anyway.


Resolving patch:
diff --git a/IPython/utils/path.py b/IPython/utils/path.py
--- a/IPython/utils/path.py
+++ b/IPython/utils/path.py
@@ -373,8 +373,11 @@ def shellglob(args):
 
     """
     expanded = []
+    # Do not unescape backslash in Windows as it is interpreted as
+    # path separator:
+    unescape = unescape_glob if sys.platform != 'win32' else lambda x: x
     for a in args:
-        expanded.extend(glob.glob(a) or [unescape_glob(a)])
+        expanded.extend(glob.glob(a) or [unescape(a)])
     return expanded
 
 


Test patch: