Repository: ipython/ipython

Issue:
Windows: regression in cd magic handling of paths
A. In 0.10.2, the user could copy a path from elsewhere, and paste it after cd magic, backslashes and all:

```
cd c:\Users\myname
```

In 0.11, this fails:
SyntaxError: (unicode error) 'rawunicodeescape' codec can't decode bytes in position 5-6: truncated \uXXXX...

No form of quoting (including raw) seems to help.

For manual path entry, forward slashes or double backslashes work, but for copy/paste this is a regression.

B. In 0.10, tab completion of quoted paths with spaces worked a bit (Tabbing immediately after a space, it failed, but then tabbing after a subsequent nonspace it worked again.) AFAICT in 0.12 there is no way to get tab completion of paths with spaces (quoted or not) working following a space character.

C. In 0.10, %cd could recognize paths which included spaces, quoted or unquoted. Not so in 0.12, which requires quotes. Arguable whether this is a troublesome regression or a logical improvement.



Discussion:
Can you show the traceback that goes with that SyntaxError?

(High priority for now because the pasting of paths would be particularly good to have)

Not much more than that:

```
File "<ipython-input-3-4d4e46418f5a>", line 1
SyntaxError: (unicode error) 'rawunicodeescape' codec can't decode bytes 
in position 5-6: truncated \uXXXX (<ipython-input-3-4d4e46418f5a>, line 1)
```

If a Windows person wants to put some time into this, that would be great.  I stared at it for a while prior to 0.11, but Windows paths were just too much of a mess.

The input gets translated OK, but when we try to parse it to an AST, Python
sees it as a raw unicode literal: `u'get_ipython().magic(ur"cd
c:\\Users\\myname")\n'` . Unicode escapes are still processed in raw
unicode literals, so I think it will fail on any folder beginning with u or
U, such as Users.

In fact, this will probably affect any magic command with the substring \u
or \U, but cd is the most common case. Perhaps we should replace \ with \,
and not construct a raw string literal. i.e. use `u"cd c:\\Users\\myname"`
rather than `ur"cd c:\Users\myname"`.

@takluyver, good point.  In fact, if it's really this that we are producing:

```
`u'get_ipython().magic(ur"cd c:\\Users\\myname")\n'`
```

then we're simply doing it wrong; it looks like we're both escaping _and_ making it a raw literal, and that's the bug.  By escaping it out we duplicate the \, but since it's raw, the second one gets interpreted as a unicode escape instead of a backslash.  We should either escape OR make it raw, but not both.

It's somewhat confusing - because the code is itself in a string, those are
actually single slashes, repr-ed as double slashes. So we're making it raw,
but not escaping.

I think we assumed at some point that making it raw removed the need to
escape. Unfortunately, that doesn't hold for unicode escapes in raw unicode
strings.

Ah, got it.  OK, then in that case it seems like we should escape \ instead of making it raw.  Do we also need to escape ' ' (space)?

Not at the same level, I don't think. We may need to inside cd, I'm not
sure.

paths with spaces in them are pure evil, but people in windows use them all the time (and XP made it mandatory, with it's boneheaded 'c:\documents and settings' or whatever that directory was named...)

It's honestly pathetic that spaces cause as much trouble as they do.  It should be considered a huge failing of command-line environments in general, if support for perfectly logical names on the filesystem is ever anything more than trivial.

FWIW Windows 7 (or possibly Vista) changed the default path, so now it's `C:\Users\username`

Well, it's the clash with the years-old convenience of using space as an argument separator.  Since the spacebar is the easiest key to hit, cmd line environments made the choice of using spaces into the default, universal argument separator (instead of commas like programming languages use).  But that meant having to escape spaces all the time.  And it means that string splitting logic must be very careful because it must avoid splitting when escapes are present.  That's why it's so easy to get code that works with spaces at one level, but once that code calls something else things break, because the escaping at one level is lost at the next...

In any case, we should make it as easy as possible for people to cd, that's for sure :)

Yes, and this is an inherent advantage of GUIs over command-line interfaces.  This is all part of why Windows is so horrible as a CLI environment - they really have no interest in it, and if they don't care about CLIs, spaces don't cause problems, so they are free to use them.  That of course causes a great deal of pain for those of us who _do_ want a sensible CLI on Windows.

Agreed.  Funnily enough, as much as I despise spaces in filenames, I find myself naturally gravitating towards them when I use a GUI for file operations; it's just a natural thing to want...

In any case, it seems like (trying to get back on topic :)  for this bug we need to at least escape \ and test what happens with ' '.  Shouldn't be too hard... This seems like a nasty regression, but I'm worried that if we don't draw the line somewhere we'll never get 0.12 out so I'm reluctant to make it critical.  If anyone can provide a PR for this soon, that would go a big ways towards getting it fixed :)

I'll look into it.

The issue is in `IPython.utils.text.make_quoted_expr`. I think it's fixable, but I suddenly wondered: is make_quoted_expr(s) actually doing anything that we couldn't achieve just by using repr? Is there any case where `eval(repr(s)) != s` (with a string s)?

@takluyver, that's a very good point!  Looking at that code, which is very old and ugly, I'm not sure we actually need it.  If you want to take a stab at just replacing that function for now with a bare `return repr(s)` and seeing how things go, it would be great.  The more of those home-brewed weird/hackish little utilities we get rid of, the better off we'll be.


Resolving patch:
diff --git a/IPython/core/inputsplitter.py b/IPython/core/inputsplitter.py
--- a/IPython/core/inputsplitter.py
+++ b/IPython/core/inputsplitter.py
@@ -75,7 +75,6 @@
 
 # IPython modules
 from IPython.core.splitinput import split_user_input, LineInfo
-from IPython.utils.text import make_quoted_expr
 from IPython.utils.py3compat import cast_unicode
 
 #-----------------------------------------------------------------------------
@@ -503,8 +502,7 @@ def transform_assign_system(line):
     if m is not None:
         cmd = m.group('cmd')
         lhs = m.group('lhs')
-        expr = make_quoted_expr(cmd)
-        new_line = '%s = get_ipython().getoutput(%s)' % (lhs, expr)
+        new_line = '%s = get_ipython().getoutput(%r)' % (lhs, cmd)
         return new_line
     return line
 
@@ -518,8 +516,7 @@ def transform_assign_magic(line):
     if m is not None:
         cmd = m.group('cmd')
         lhs = m.group('lhs')
-        expr = make_quoted_expr(cmd)
-        new_line = '%s = get_ipython().magic(%s)' % (lhs, expr)
+        new_line = '%s = get_ipython().magic(%r)' % (lhs, cmd)
         return new_line
     return line
 
@@ -560,13 +557,13 @@ def _make_help_call(target, esc, lspace, next_input=None):
     method  = 'pinfo2' if esc == '??' \
                 else 'psearch' if '*' in target \
                 else 'pinfo'
-    arg = make_quoted_expr(" ".join([method, target]))
+    arg = " ".join([method, target])
     
     if next_input:
-        tpl = '%sget_ipython().magic(%s, next_input=%s)'
-        return tpl % (lspace, arg, make_quoted_expr(next_input))
+        tpl = '%sget_ipython().magic(%r, next_input=%r)'
+        return tpl % (lspace, arg, next_input)
     else:
-        return '%sget_ipython().magic(%s)' % (lspace, arg)
+        return '%sget_ipython().magic(%r)' % (lspace, arg)
 
 _initial_space_re = re.compile(r'\s*')
 _help_end_re = re.compile(r"""(%?
@@ -610,15 +607,13 @@ def __init__(self):
     def _tr_system(line_info):
         "Translate lines escaped with: !"
         cmd = line_info.line.lstrip().lstrip(ESC_SHELL)
-        return '%sget_ipython().system(%s)' % (line_info.pre,
-                                               make_quoted_expr(cmd))
+        return '%sget_ipython().system(%r)' % (line_info.pre, cmd)
 
     @staticmethod
     def _tr_system2(line_info):
         "Translate lines escaped with: !!"
         cmd = line_info.line.lstrip()[2:]
-        return '%sget_ipython().getoutput(%s)' % (line_info.pre,
-                                                  make_quoted_expr(cmd))
+        return '%sget_ipython().getoutput(%r)' % (line_info.pre, cmd)
 
     @staticmethod
     def _tr_help(line_info):
@@ -632,9 +627,8 @@ def _tr_help(line_info):
     @staticmethod
     def _tr_magic(line_info):
         "Translate lines escaped with: %"
-        tpl = '%sget_ipython().magic(%s)'
-        cmd = make_quoted_expr(' '.join([line_info.ifun,
-                                        line_info.the_rest]).strip())
+        tpl = '%sget_ipython().magic(%r)'
+        cmd = ' '.join([line_info.ifun, line_info.the_rest]).strip()
         return tpl % (line_info.pre, cmd)
 
     @staticmethod
diff --git a/IPython/core/prefilter.py b/IPython/core/prefilter.py
--- a/IPython/core/prefilter.py
+++ b/IPython/core/prefilter.py
@@ -36,7 +36,6 @@
 from IPython.core import page
 
 from IPython.utils.traitlets import List, Integer, Any, Unicode, CBool, Bool, Instance
-from IPython.utils.text import make_quoted_expr
 from IPython.utils.autoattr import auto_attr
 
 #-----------------------------------------------------------------------------
@@ -405,8 +404,8 @@ def transform(self, line, continue_prompt):
         if m is not None:
             cmd = m.group('cmd')
             lhs = m.group('lhs')
-            expr = make_quoted_expr("sc =%s" % cmd)
-            new_line = '%s = get_ipython().magic(%s)' % (lhs, expr)
+            expr = "sc =%s" % cmd
+            new_line = '%s = get_ipython().magic(%r)' % (lhs, expr)
             return new_line
         return line
 
@@ -424,8 +423,7 @@ def transform(self, line, continue_prompt):
         if m is not None:
             cmd = m.group('cmd')
             lhs = m.group('lhs')
-            expr = make_quoted_expr(cmd)
-            new_line = '%s = get_ipython().magic(%s)' % (lhs, expr)
+            new_line = '%s = get_ipython().magic(%r)' % (lhs, cmd)
             return new_line
         return line
 
@@ -733,8 +731,7 @@ def handle(self, line_info):
         transformed = self.shell.alias_manager.expand_aliases(line_info.ifun,line_info.the_rest)
         # pre is needed, because it carries the leading whitespace.  Otherwise
         # aliases won't work in indented sections.
-        line_out = '%sget_ipython().system(%s)' % (line_info.pre_whitespace,
-                                         make_quoted_expr(transformed))
+        line_out = '%sget_ipython().system(%r)' % (line_info.pre_whitespace, transformed)
 
         return line_out
 
@@ -762,8 +759,7 @@ def handle(self, line_info):
             return magic_handler.handle(line_info)
         else:
             cmd = line.lstrip().lstrip(ESC_SHELL)
-            line_out = '%sget_ipython().system(%s)' % (line_info.pre_whitespace,
-                                             make_quoted_expr(cmd))
+            line_out = '%sget_ipython().system(%r)' % (line_info.pre_whitespace, cmd)
         return line_out
 
 
@@ -786,8 +782,8 @@ def handle(self, line_info):
         """Execute magic functions."""
         ifun    = line_info.ifun
         the_rest = line_info.the_rest
-        cmd = '%sget_ipython().magic(%s)' % (line_info.pre_whitespace,
-                                   make_quoted_expr(ifun + " " + the_rest))
+        cmd = '%sget_ipython().magic(%r)' % (line_info.pre_whitespace,
+                                                    (ifun + " " + the_rest))
         return cmd
 
 
diff --git a/IPython/utils/text.py b/IPython/utils/text.py
--- a/IPython/utils/text.py
+++ b/IPython/utils/text.py
@@ -295,40 +295,6 @@ def esc_quotes(strng):
     return strng.replace('"','\\"').replace("'","\\'")
 
 
-def make_quoted_expr(s):
-    """Return string s in appropriate quotes, using raw string if possible.
-
-    XXX - example removed because it caused encoding errors in documentation
-    generation.  We need a new example that doesn't contain invalid chars.
-
-    Note the use of raw string and padding at the end to allow trailing
-    backslash.
-    """
-
-    tail = ''
-    tailpadding = ''
-    raw  = ''
-    ucode = '' if py3compat.PY3 else 'u'
-    if "\\" in s:
-        raw = 'r'
-        if s.endswith('\\'):
-            tail = '[:-1]'
-            tailpadding = '_'
-    if '"' not in s:
-        quote = '"'
-    elif "'" not in s:
-        quote = "'"
-    elif '"""' not in s and not s.endswith('"'):
-        quote = '"""'
-    elif "'''" not in s and not s.endswith("'"):
-        quote = "'''"
-    else:
-        # give up, backslash-escaped string will do
-        return '"%s"' % esc_quotes(s)
-    res = ucode + raw + quote + s + tailpadding + quote + tail
-    return res
-
-
 def qw(words,flat=0,sep=None,maxsplit=-1):
     """Similar to Perl's qw() operator, but with some more options.
 


Test patch:
diff --git a/IPython/config/tests/test_configurable.py b/IPython/config/tests/test_configurable.py
--- a/IPython/config/tests/test_configurable.py
+++ b/IPython/config/tests/test_configurable.py
@@ -31,7 +31,7 @@
 )
 
 from IPython.config.loader import Config
-
+from IPython.utils.py3compat import PY3
 
 #-----------------------------------------------------------------------------
 # Test cases
@@ -62,6 +62,11 @@ class MyConfigurable(Configurable):
     Current: 4.0
     The integer b."""
 
+# On Python 3, the Integer trait is a synonym for Int
+if PY3:
+    mc_help = mc_help.replace(u"<Integer>", u"<Int>")
+    mc_help_inst = mc_help_inst.replace(u"<Integer>", u"<Int>")
+
 class Foo(Configurable):
     a = Integer(0, config=True, help="The integer a.")
     b = Unicode('nope', config=True)
diff --git a/IPython/core/tests/test_handlers.py b/IPython/core/tests/test_handlers.py
--- a/IPython/core/tests/test_handlers.py
+++ b/IPython/core/tests/test_handlers.py
@@ -61,10 +61,10 @@ def test_handlers():
     ip.alias_manager.alias_table['top'] = (0, 'd:/cygwin/top')
     ip.alias_manager.alias_table['d'] =   (0, 'true')
     run([(i,py3compat.u_format(o)) for i,o in \
-        [("an_alias",    'get_ipython().system({u}"true ")'),     # alias
+        [("an_alias",    "get_ipython().system({u}'true ')"),     # alias
          # Below: recursive aliases should expand whitespace-surrounded
          # chars, *not* initial chars which happen to be aliases:
-         ("top",         'get_ipython().system({u}"d:/cygwin/top ")'),
+         ("top",         "get_ipython().system({u}'d:/cygwin/top ')"),
          ]])
     ip.system = old_system_cmd
 
@@ -76,15 +76,15 @@ def test_handlers():
     # line.
     run([(i,py3compat.u_format(o)) for i,o in \
         [('"no change"', '"no change"'),             # normal
-         ("!true",       'get_ipython().system({u}"true")'),      # shell_escapes
-         ("!! true",     'get_ipython().magic({u}"sx  true")'),   # shell_escapes + magic
-         ("!!true",      'get_ipython().magic({u}"sx true")'),    # shell_escapes + magic
-         ("%lsmagic",    'get_ipython().magic({u}"lsmagic ")'),   # magic
-         ("lsmagic",     'get_ipython().magic({u}"lsmagic ")'),   # magic
+         (u"!true",       "get_ipython().system({u}'true')"),      # shell_escapes
+         (u"!! true",     "get_ipython().magic({u}'sx  true')"),   # shell_escapes + magic
+         (u"!!true",      "get_ipython().magic({u}'sx true')"),    # shell_escapes + magic
+         (u"%lsmagic",    "get_ipython().magic({u}'lsmagic ')"),   # magic
+         (u"lsmagic",     "get_ipython().magic({u}'lsmagic ')"),   # magic
          #("a = b # PYTHON-MODE", '_i'),          # emacs -- avoids _in cache
 
          # post-esc-char whitespace goes inside
-         ("! true",   'get_ipython().system({u}" true")'),
+         (u"! true",   "get_ipython().system({u}' true')"),
 
          # handle_help
 
@@ -99,19 +99,19 @@ def test_handlers():
     ip.prefilter_manager.multi_line_specials = False
     # W/ multi_line_specials off, leading ws kills esc chars/autoexpansion
     run([
-        ('if 1:\n    !true',    'if 1:\n    !true'),
-        ('if 1:\n    lsmagic',  'if 1:\n    lsmagic'),
-        ('if 1:\n    an_alias', 'if 1:\n    an_alias'),
+        (u'if 1:\n    !true',    u'if 1:\n    !true'),
+        (u'if 1:\n    lsmagic',  u'if 1:\n    lsmagic'),
+        (u'if 1:\n    an_alias', u'if 1:\n    an_alias'),
         ])
 
     ip.prefilter_manager.multi_line_specials = True
     # initial indents must be preserved.
     run([(i,py3compat.u_format(o)) for i,o in \
-        [('if 1:\n    !true',    'if 1:\n    get_ipython().system({u}"true")'),
-         ('if 2:\n    lsmagic',  'if 2:\n    get_ipython().magic({u}"lsmagic ")'),
-         ('if 1:\n    an_alias', 'if 1:\n    get_ipython().system({u}"true ")'),
+        [(u'if 1:\n    !true',    "if 1:\n    get_ipython().system({u}'true')"),
+         (u'if 2:\n    lsmagic',  "if 2:\n    get_ipython().magic({u}'lsmagic ')"),
+         (u'if 1:\n    an_alias', "if 1:\n    get_ipython().system({u}'true ')"),
          # Weird one
-         ('if 1:\n    !!true',   'if 1:\n    get_ipython().magic({u}"sx true")'),
+         (u'if 1:\n    !!true',   "if 1:\n    get_ipython().magic({u}'sx true')"),
 
          # Even with m_l_s on, autocall is off even with special chars
          ('if 1:\n    /fun 1 2', 'if 1:\n    /fun 1 2'),
diff --git a/IPython/core/tests/test_inputsplitter.py b/IPython/core/tests/test_inputsplitter.py
--- a/IPython/core/tests/test_inputsplitter.py
+++ b/IPython/core/tests/test_inputsplitter.py
@@ -405,16 +405,16 @@ def transform_checker(tests, func):
 syntax = \
   dict(assign_system =
        [(i,py3compat.u_format(o)) for i,o in \
-       [('a =! ls', 'a = get_ipython().getoutput({u}"ls")'),
-        ('b = !ls', 'b = get_ipython().getoutput({u}"ls")'),
+       [(u'a =! ls', "a = get_ipython().getoutput({u}'ls')"),
+        (u'b = !ls', "b = get_ipython().getoutput({u}'ls')"),
         ('x=1', 'x=1'), # normal input is unmodified
         ('    ','    '),  # blank lines are kept intact
         ]],
 
        assign_magic =
        [(i,py3compat.u_format(o)) for i,o in \
-       [('a =% who', 'a = get_ipython().magic({u}"who")'),
-        ('b = %who', 'b = get_ipython().magic({u}"who")'),
+       [(u'a =% who', "a = get_ipython().magic({u}'who')"),
+        (u'b = %who', "b = get_ipython().magic({u}'who')"),
         ('x=1', 'x=1'), # normal input is unmodified
         ('    ','    '),  # blank lines are kept intact
         ]],
@@ -442,43 +442,45 @@ def transform_checker(tests, func):
        # System calls
        escaped_shell =
        [(i,py3compat.u_format(o)) for i,o in \
-       [ ('!ls', 'get_ipython().system({u}"ls")'),
+       [ (u'!ls', "get_ipython().system({u}'ls')"),
          # Double-escape shell, this means to capture the output of the
          # subprocess and return it
-         ('!!ls', 'get_ipython().getoutput({u}"ls")'),
+         (u'!!ls', "get_ipython().getoutput({u}'ls')"),
          ]],
 
        # Help/object info
        escaped_help =
        [(i,py3compat.u_format(o)) for i,o in \
-       [ ('?', 'get_ipython().show_usage()'),
-         ('?x1', 'get_ipython().magic({u}"pinfo x1")'),
-         ('??x2', 'get_ipython().magic({u}"pinfo2 x2")'),
-         ('?a.*s', 'get_ipython().magic({u}"psearch a.*s")'),
-         ('?%hist', 'get_ipython().magic({u}"pinfo %hist")'),
-         ('?abc = qwe', 'get_ipython().magic({u}"pinfo abc")'),
+       [ (u'?', 'get_ipython().show_usage()'),
+         (u'?x1', "get_ipython().magic({u}'pinfo x1')"),
+         (u'??x2', "get_ipython().magic({u}'pinfo2 x2')"),
+         (u'?a.*s', "get_ipython().magic({u}'psearch a.*s')"),
+         (u'?%hist', "get_ipython().magic({u}'pinfo %hist')"),
+         (u'?abc = qwe', "get_ipython().magic({u}'pinfo abc')"),
          ]],
 
       end_help =
       [(i,py3compat.u_format(o)) for i,o in \
-      [ ('x3?', 'get_ipython().magic({u}"pinfo x3")'),
-        ('x4??', 'get_ipython().magic({u}"pinfo2 x4")'),
-        ('%hist?', 'get_ipython().magic({u}"pinfo %hist")'),
-        ('f*?', 'get_ipython().magic({u}"psearch f*")'),
-        ('ax.*aspe*?', 'get_ipython().magic({u}"psearch ax.*aspe*")'),
-        ('a = abc?', 'get_ipython().magic({u}"pinfo abc", next_input={u}"a = abc")'),
-        ('a = abc.qe??', 'get_ipython().magic({u}"pinfo2 abc.qe", next_input={u}"a = abc.qe")'),
-        ('a = *.items?', 'get_ipython().magic({u}"psearch *.items", next_input={u}"a = *.items")'),
-        ('plot(a?', 'get_ipython().magic({u}"pinfo a", next_input={u}"plot(a")'),
-        ('a*2 #comment?', 'a*2 #comment?'),
+      [ (u'x3?', "get_ipython().magic({u}'pinfo x3')"),
+        (u'x4??', "get_ipython().magic({u}'pinfo2 x4')"),
+        (u'%hist?', "get_ipython().magic({u}'pinfo %hist')"),
+        (u'f*?', "get_ipython().magic({u}'psearch f*')"),
+        (u'ax.*aspe*?', "get_ipython().magic({u}'psearch ax.*aspe*')"),
+        (u'a = abc?', "get_ipython().magic({u}'pinfo abc', next_input={u}'a = abc')"),
+        (u'a = abc.qe??', "get_ipython().magic({u}'pinfo2 abc.qe', next_input={u}'a = abc.qe')"),
+        (u'a = *.items?', "get_ipython().magic({u}'psearch *.items', next_input={u}'a = *.items')"),
+        (u'plot(a?', "get_ipython().magic({u}'pinfo a', next_input={u}'plot(a')"),
+        (u'a*2 #comment?', 'a*2 #comment?'),
         ]],
 
        # Explicit magic calls
        escaped_magic =
        [(i,py3compat.u_format(o)) for i,o in \
-       [ ('%cd', 'get_ipython().magic({u}"cd")'),
-         ('%cd /home', 'get_ipython().magic({u}"cd /home")'),
-         ('    %magic', '    get_ipython().magic({u}"magic")'),
+       [ (u'%cd', "get_ipython().magic({u}'cd')"),
+         (u'%cd /home', "get_ipython().magic({u}'cd /home')"),
+         # Backslashes need to be escaped.
+         (u'%cd C:\\User', "get_ipython().magic({u}'cd C:\\\\User')"),
+         (u'    %magic', "    get_ipython().magic({u}'magic')"),
          ]],
 
        # Quoting with separate arguments
@@ -508,11 +510,11 @@ def transform_checker(tests, func):
        # Check that we transform prompts before other transforms
        mixed =
        [(i,py3compat.u_format(o)) for i,o in \
-       [ ('In [1]: %lsmagic', 'get_ipython().magic({u}"lsmagic")'),
-         ('>>> %lsmagic', 'get_ipython().magic({u}"lsmagic")'),
-         ('In [2]: !ls', 'get_ipython().system({u}"ls")'),
-         ('In [3]: abs?', 'get_ipython().magic({u}"pinfo abs")'),
-         ('In [4]: b = %who', 'b = get_ipython().magic({u}"who")'),
+       [ (u'In [1]: %lsmagic', "get_ipython().magic({u}'lsmagic')"),
+         (u'>>> %lsmagic', "get_ipython().magic({u}'lsmagic')"),
+         (u'In [2]: !ls', "get_ipython().system({u}'ls')"),
+         (u'In [3]: abs?', "get_ipython().magic({u}'pinfo abs')"),
+         (u'In [4]: b = %who', "b = get_ipython().magic({u}'who')"),
          ]],
        )