Repository: ipython/ipython

Issue:
Unicode bug in Itpl when expanding shell variables in syscalls with !
found this today during a presentation, dumping quickly...

<pre>
In [35]: files
Out[35]: 
['pics:',
 '83547885.jpg',
 '',
 'ppt:',
 'Aiguille du Midi.pps',
 'Ca\xc3\xb1o Cristales.pps',
 'parejas disparejas.ppt',
 'Underwater.pps',
 '',
 'pub:',
 'image_summary.py',
 'stained_glass_barcelona.png',
 'trapezoid_demo.py',
 'trapezoid.py',
 'trap.py',
 'trap.py~']

In [36]: for f in files:
   ....:     !echo $f
   ....:     
pics:
83547885.jpg

ppt:
Aiguille du Midi.pps
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
/home/fperez/tmp/<ipython-input-36-df6db386b8e9> in <module>()
      1 for f in files:
----> 2     get_ipython().system(u"echo $f")
      3 

/home/fperez/usr/lib/python2.6/site-packages/IPython/core/interactiveshell.pyc in system_raw(self, cmd)
   1999         # a non-None value would trigger :func:`sys.displayhook` calls.

   2000         # Instead, we store the exit_code in user_ns.

-> 2001         self.user_ns['_exit_code'] = os.system(self.var_expand(cmd, depth=2))
   2002 
   2003     # use piped system by default, because it is better behaved


/home/fperez/usr/lib/python2.6/site-packages/IPython/core/interactiveshell.pyc in var_expand(self, cmd, depth)
   2473                           sys._getframe(depth+1).f_locals # locals
   2474                           )
-> 2475         return py3compat.str_to_unicode(str(res), res.codec)
   2476 
   2477     def mktempfile(self, data=None, prefix='ipython_edit_'):

/home/fperez/usr/lib/python2.6/site-packages/IPython/external/Itpl/_Itpl.pyc in __str__(self)
    240     def __str__(self):
    241         """Evaluate and substitute the appropriate parts of the string."""
--> 242         return self._str(self.globals,self.locals)
    243 
    244     def __repr__(self):

/home/fperez/usr/lib/python2.6/site-packages/IPython/external/Itpl/_Itpl.pyc in _str(self, glob, loc)
    197             if live: app(str(eval(chunk,glob,loc)))
    198             else: app(chunk)
--> 199         out = ''.join(result)
    200         try:
    201             return str(out)

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2: ordinal not in range(128)
</pre>



Discussion:
Isn't it well established that Itpl just doesn't support unicode in the slightest?

Right... I had people in front of me so it was really quick and I didn't have the bandwidth to recall that.

One more reason to push on moving out of Itpl, then.  Unicode in the filesystem is more and more common, so this will be more annoying to people as time goes on.  I don't know if we'll manage to transition it by 0.12 though.

Just to mention, the `FullEvalFormatter` that I'd like to use for this is in PR #507 (prompt manager). So I'd prefer to get that merged in before approaching this. But if the prompt manager is going to be delayed until after 0.12, I can copy `FullEvalFormatter` across and do this first.

@fperez: Is the prompt manager stuff likely to get in for 0.12? If not, I can copy across the `FullEvalFormatter` code and replace Itpl here with string formatting.

Let's see if we can find the time to resolve those.  If we can make headway into the harder PRs we have before release, we won't need to.  I'd like to tackle them one at a time over the next week, we'll see how it goes.

Itpl is actually quite small, and it was easy to get it working, at least for this particular bug.  If we do want to keep it, should I keep it str-only, or make it unicode-native, or just leave it like Thomas says, and move to EvalFormatter (losing '$' in the process).

EvalFormatter is definitely much better (and more pythonic) code.  But I have a feeling that the people who use the '$' expansion will be very sad to see it gone.  Especially if we restore the long lost shell profile.

I'd forgotten that itpl is the one  that gives us $ expansion, and that is something that is _definitely_ very useful, and that I've demoed multiple times to audiences in addition to using it personally quite regularly.

So that's a good argument for keeping Itpl around then, even if we move to EvalFormatter for most of our internal use.  In that case, making it unicode compliant seems like the right way to go, as that will ensure things work OK in py3.

Making $name and $name.attr work should just be a few lines subclass of FullEvalFormatter, so if it allows us to drop ~300 lines of Itpl (which it seems we now need to maintain ourselves), I think it's worth doing.

More complex expressions will need to be written as `${name['item'](args)}`, but I don't think that's a show stopper.

On Wed, Oct 19, 2011 at 5:10 AM, Thomas
reply@reply.github.com
wrote:

> Making $name and $name.attr work should just be a few lines subclass of FullEvalFormatter, so if it allows us to drop ~300 lines of Itpl (which it seems we now need to maintain ourselves), I think it's worth doing.
> 
> More complex expressions will need to be written as `${name['item'](args)}`, but I don't think that's a show stopper.

+1.  I think $n and $n.a are the two main use cases for this, so I'm all for it.

Great - I'll get onto this either when the prompt manager stuff gets merged,
or if we decide that's getting deferred until 0.13.

On Wed, Oct 19, 2011 at 9:15 AM, Thomas
reply@reply.github.com
wrote:

> Great - I'll get onto this either when the prompt manager stuff gets merged,
> or if we decide that's getting deferred until 0.13.

OK.  I've been trying to get to the 'meaty' PRs, but the flow of
lightweight ones just won't stop :)  Soon we're going to have to
perhaps switch strategies and tackle the meaty ones even if it means
allowing quick ones to pile up, else we'll never clear those which
require more thought and effort, and that I want to have in before
they bit-rot.

Sounds good - I'll slow down on the small ones, I was just trying to clean out some of the easy 0.12 issues.

Thomas, how were you thinking adding '$foo' support would work in FullEval?  All the actual parsing in handled by the `str._formatter_parser` method, so it seems like you would essentially have to rewrite the Itpl parse all over again.

I should also note that when I was digging into this ( I do already have unicode itpl working ), I discovered a small related bug - os.system doesn't like unicode, so we have to make sure that we encode with unicode_to_str when we pass to it.

On Wed, Oct 19, 2011 at 10:00 AM, Min RK
reply@reply.github.com
wrote:

> Sounds good - I'll slow down on the small ones, I was just trying to clean out some of the easy 0.12 issues.

I wasn't trying to sound critical, BTW.  It's _awesome_ that you
flushed so many of those, and that's why I made an effort to keep up
on the receiving end.  So thanks a ton for all that work: in the
aggregate it's actually a lot of significant improvements.  But I do
worry about some of the harder ones at the bottom of the PR list.  The
longer they linger, the higher the chances of merge conflicts that may
be non-trivial to resolve, and also that we simply forget the context
and details of the discussion (happens to me at least).

On 19 October 2011 18:00, Min RK <
reply@reply.github.com>wrote:

> Thomas, how were you thinking adding '$foo' support would work in FullEval?
>  All the actual parsing in handled by the `str._formatter_parser` method, so
> it seems like you would essentially have to rewrite the Itpl parse all over
> again.

Itpl does character by character processing, so you can have complex
expressions with just a $ prefix. But I think, as Fernando says, most of the
usage is the simple cases - $name and $name.attr. If it's OK to require {}
braces around any more complex expressions, then I think we can get by with
a simple regex, and improve performance while we're at it.

string.Formatter has an overridable .parse method, where this logic can go.

okay, makes sense.

On Wed, Oct 19, 2011 at 10:56 AM, Thomas
reply@reply.github.com
wrote:

> If it's OK to require {}
> braces around any more complex expressions, then I think we can get by with
> a simple regex, and improve performance while we're at it.

It's OK also b/c I think it improves readability.  For a complex
expression, the braces help the reader clearly understand what the
expression is, without having to guess the behavior of the underlying
parser (and/or its bugs).  So $x, $x.a, ${complicated expr}, seems
like a perfect design brief moving forward (even if it's not precisely
what itpl did historically).

So am I correct in understanding that the official plan for this is for Thomas to add `$foo` support to FullEvalFormatter, and remove Itpl as part of the PromptManager PR?

On Fri, Oct 21, 2011 at 2:23 PM, Min RK
reply@reply.github.com
wrote:

> So am I correct in understanding that the official plan for this is for Thomas to add `$foo` support to FullEvalFormatter, and remove Itpl as part of the PromptManager PR?

At least moving over to FullEvalFormatter.  I'd hold off  on removing
Itpl altogether for a release or two, just in case people use it
through us.  But we can stop using it and slap a deprecation warning
on it so anyone importing it from us gets a heads-up.


Resolving patch:
diff --git a/IPython/core/interactiveshell.py b/IPython/core/interactiveshell.py
--- a/IPython/core/interactiveshell.py
+++ b/IPython/core/interactiveshell.py
@@ -73,7 +73,8 @@
 from IPython.utils.process import system, getoutput
 from IPython.utils.strdispatch import StrDispatch
 from IPython.utils.syspathcontext import prepended_to_syspath
-from IPython.utils.text import num_ini_spaces, format_screen, LSString, SList
+from IPython.utils.text import (num_ini_spaces, format_screen, LSString, SList,
+                                DollarFormatter)
 from IPython.utils.traitlets import (Integer, CBool, CaselessStrEnum, Enum,
                                      List, Unicode, Instance, Type)
 from IPython.utils.warn import warn, error, fatal
@@ -2571,7 +2572,7 @@ def enable_pylab(self, gui=None, import_all=True):
     # Utilities
     #-------------------------------------------------------------------------
 
-    def var_expand(self,cmd,depth=0):
+    def var_expand(self, cmd, depth=0, formatter=DollarFormatter()):
         """Expand python variables in a string.
 
         The depth argument indicates how many frames above the caller should
@@ -2580,11 +2581,10 @@ def var_expand(self,cmd,depth=0):
         The global namespace for expansion is always the user's interactive
         namespace.
         """
-        res = ItplNS(cmd, self.user_ns,  # globals
-                          # Skip our own frame in searching for locals:
-                          sys._getframe(depth+1).f_locals # locals
-                          )
-        return py3compat.str_to_unicode(str(res), res.codec)
+        ns = self.user_ns.copy()
+        ns.update(sys._getframe(depth+1).f_locals)
+        ns.pop('self', None)
+        return formatter.format(cmd, **ns)
 
     def mktempfile(self, data=None, prefix='ipython_edit_'):
         """Make a new tempfile and return its filename.
diff --git a/IPython/utils/text.py b/IPython/utils/text.py
--- a/IPython/utils/text.py
+++ b/IPython/utils/text.py
@@ -25,6 +25,7 @@
 from string import Formatter
 
 from IPython.external.path import path
+from IPython.testing.skipdoctest import skip_doctest_py3
 from IPython.utils import py3compat
 from IPython.utils.io import nlprint
 from IPython.utils.data import flatten
@@ -597,33 +598,53 @@ def wrap_paragraphs(text, ncols=80):
     return out_ps
 
 
-
 class EvalFormatter(Formatter):
     """A String Formatter that allows evaluation of simple expressions.
-
-    Any time a format key is not found in the kwargs,
-    it will be tried as an expression in the kwargs namespace.
-
+    
+    Note that this version interprets a : as specifying a format string (as per
+    standard string formatting), so if slicing is required, you must explicitly
+    create a slice.
+    
     This is to be used in templating cases, such as the parallel batch
     script templates, where simple arithmetic on arguments is useful.
 
     Examples
     --------
+    
+    In  [1]: f = EvalFormatter()
+    In  [2]: f.format('{n//4}', n=8)
+    Out [2]: '2'
+    
+    In  [3]: f.format("{greeting[slice(2,4)]}", greeting="Hello")
+    Out [3]: 'll'
+    """
+    def get_field(self, name, args, kwargs):
+        v = eval(name, kwargs)
+        return v, name
 
-    In [1]: f = EvalFormatter()
+@skip_doctest_py3
+class FullEvalFormatter(Formatter):
+    """A String Formatter that allows evaluation of simple expressions.
+    
+    Any time a format key is not found in the kwargs,
+    it will be tried as an expression in the kwargs namespace.
+    
+    Note that this version allows slicing using [1:2], so you cannot specify
+    a format string. Use :class:`EvalFormatter` to permit format strings.
+    
+    Examples
+    --------
+    
+    In [1]: f = FullEvalFormatter()
     In [2]: f.format('{n//4}', n=8)
-    Out[2]: '2'
-
-    In [3]: f.format('{list(range(3))}')
-    Out[3]: '[0, 1, 2]'
+    Out[2]: u'2'
+    
+    In [3]: f.format('{list(range(5))[2:4]}')
+    Out[3]: u'[2, 3]'
 
     In [4]: f.format('{3*2}')
-    Out[4]: '6'
+    Out[4]: u'6'
     """
-
-    # should we allow slicing by disabling the format_spec feature?
-    allow_slicing = True
-
     # copied from Formatter._vformat with minor changes to allow eval
     # and replace the format_spec code with slicing
     def _vformat(self, format_string, args, kwargs, used_args, recursion_depth):
@@ -640,12 +661,11 @@ def _vformat(self, format_string, args, kwargs, used_args, recursion_depth):
             # if there's a field, output it
             if field_name is not None:
                 # this is some markup, find the object and do
-                #  the formatting
+                # the formatting
 
-                if self.allow_slicing and format_spec:
+                if format_spec:
                     # override format spec, to allow slicing:
                     field_name = ':'.join([field_name, format_spec])
-                    format_spec = ''
 
                 # eval the contents of the field for the object
                 # to be formatted
@@ -654,14 +674,43 @@ def _vformat(self, format_string, args, kwargs, used_args, recursion_depth):
                 # do any conversion on the resulting object
                 obj = self.convert_field(obj, conversion)
 
-                # expand the format spec, if needed
-                format_spec = self._vformat(format_spec, args, kwargs,
-                                            used_args, recursion_depth-1)
-
                 # format the object and append to the result
-                result.append(self.format_field(obj, format_spec))
+                result.append(self.format_field(obj, ''))
+
+        return u''.join(py3compat.cast_unicode(s) for s in result)
 
-        return ''.join(result)
+@skip_doctest_py3
+class DollarFormatter(FullEvalFormatter):
+    """Formatter allowing Itpl style $foo replacement, for names and attribute
+    access only. Standard {foo} replacement also works, and allows full
+    evaluation of its arguments.
+
+    Examples
+    --------
+    In [1]: f = DollarFormatter()
+    In [2]: f.format('{n//4}', n=8)
+    Out[2]: u'2'
+    
+    In [3]: f.format('23 * 76 is $result', result=23*76)
+    Out[3]: u'23 * 76 is 1748'
+    
+    In [4]: f.format('$a or {b}', a=1, b=2)
+    Out[4]: u'1 or 2'
+    """
+    _dollar_pattern = re.compile("(.*)\$([\w\.]+)")
+    def parse(self, fmt_string):
+        for literal_txt, field_name, format_spec, conversion \
+                    in Formatter.parse(self, fmt_string):
+            
+            # Find $foo patterns in the literal text.
+            continue_from = 0
+            for m in self._dollar_pattern.finditer(literal_txt):
+                new_txt, new_field = m.group(1,2)
+                yield (new_txt, new_field, "", None)
+                continue_from = m.end()
+            
+            # Re-yield the {foo} style pattern
+            yield (literal_txt[continue_from:], field_name, format_spec, conversion)
 
 
 def columnize(items, separator='  ', displaywidth=80):


Test patch:
diff --git a/IPython/utils/tests/test_text.py b/IPython/utils/tests/test_text.py
--- a/IPython/utils/tests/test_text.py
+++ b/IPython/utils/tests/test_text.py
@@ -44,9 +44,8 @@ def test_columnize_long():
     out = text.columnize(items, displaywidth=size-1)
     nt.assert_equals(out, '\n'.join(items+['']))
 
-def test_eval_formatter():
-    f = text.EvalFormatter()
-    ns = dict(n=12, pi=math.pi, stuff='hello there', os=os)
+def eval_formatter_check(f):
+    ns = dict(n=12, pi=math.pi, stuff='hello there', os=os, u=u"café", b="café")
     s = f.format("{n} {n//4} {stuff.split()[0]}", **ns)
     nt.assert_equals(s, "12 3 hello")
     s = f.format(' '.join(['{n//%i}'%i for i in range(1,8)]), **ns)
@@ -58,12 +57,15 @@ def test_eval_formatter():
     s = f.format("{stuff!r}", **ns)
     nt.assert_equals(s, repr(ns['stuff']))
     
+    # Check with unicode:
+    s = f.format("{u}", **ns)
+    nt.assert_equals(s, ns['u'])
+    # This decodes in a platform dependent manner, but it shouldn't error out
+    s = f.format("{b}", **ns)
+        
     nt.assert_raises(NameError, f.format, '{dne}', **ns)
 
-
-def test_eval_formatter_slicing():
-    f = text.EvalFormatter()
-    f.allow_slicing = True
+def eval_formatter_slicing_check(f):
     ns = dict(n=12, pi=math.pi, stuff='hello there', os=os)
     s = f.format(" {stuff.split()[:]} ", **ns)
     nt.assert_equals(s, " ['hello', 'there'] ")
@@ -75,9 +77,7 @@ def test_eval_formatter_slicing():
     nt.assert_raises(SyntaxError, f.format, "{n:x}", **ns)
     
 
-def test_eval_formatter_no_slicing():
-    f = text.EvalFormatter()
-    f.allow_slicing = False
+def eval_formatter_no_slicing_check(f):
     ns = dict(n=12, pi=math.pi, stuff='hello there', os=os)
     
     s = f.format('{n:x} {pi**2:+f}', **ns)
@@ -85,3 +85,25 @@ def test_eval_formatter_no_slicing():
     
     nt.assert_raises(SyntaxError, f.format, "{a[:]}")
 
+def test_eval_formatter():
+    f = text.EvalFormatter()
+    eval_formatter_check(f)
+    eval_formatter_no_slicing_check(f)
+
+def test_full_eval_formatter():
+    f = text.FullEvalFormatter()
+    eval_formatter_check(f)
+    eval_formatter_slicing_check(f)
+
+def test_dollar_formatter():
+    f = text.DollarFormatter()
+    eval_formatter_check(f)
+    eval_formatter_slicing_check(f)
+    
+    ns = dict(n=12, pi=math.pi, stuff='hello there', os=os)
+    s = f.format("$n", **ns)
+    nt.assert_equals(s, "12")
+    s = f.format("$n.real", **ns)
+    nt.assert_equals(s, "12")
+    s = f.format("$n/{stuff[:5]}", **ns)
+    nt.assert_equals(s, "12/hello")