Repository: ipython/ipython

Issue:
Magic %paste error
off git master

```

In [2]: paste
from StringIO import StringIO

text = """a,b,c,d
one,1,2,3
one,1,2,3
,1,2,3
one,1,2,3
,1,2,3
,1,2,3
one,1,2,3
two,1,2,3"""

data = StringIO(text)
## -- End pasted text --
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/home/wesm/code/pandas/<ipython-input-2-84f91d8a6054> in <module>()
----> 1 get_ipython().magic(u'paste ')

/home/wesm/code/repos/ipython/IPython/core/interactiveshell.pyc in magic(self, arg_s, next_input)
   1996                 self._magic_locals = sys._getframe(1).f_locals
   1997             with self.builtin_trap:
-> 1998                 result = fn(magic_args)
   1999             # Ensure we're not keeping object references around:

   2000             self._magic_locals = {}

/home/wesm/code/repos/ipython/IPython/frontend/terminal/interactiveshell.pyc in magic_paste(self, parameter_s)
    651             write("## -- End pasted text --\n")
    652 
--> 653         store_or_execute(self.shell, block, name)
    654 
    655     # Class-level: add a '%cls' magic only on Windows


/home/wesm/code/repos/ipython/IPython/frontend/terminal/interactiveshell.pyc in store_or_execute(shell, block, name)
     93     # Dedent and prefilter so what we store matches what is executed by

     94     # run_cell.

---> 95     b = shell.prefilter(textwrap.dedent(block))
     96 
     97     if name:

/home/wesm/code/repos/ipython/IPython/core/prefilter.pyc in prefilter_lines(self, lines, continue_prompt)
    355         if len(llines) > 1:
    356             out = '\n'.join([self.prefilter_line(line, lnum>0)
--> 357                              for lnum, line in enumerate(llines) ])
    358         else:
    359             out = self.prefilter_line(llines[0], continue_prompt)

/home/wesm/code/repos/ipython/IPython/core/prefilter.pyc in prefilter_line(self, line, continue_prompt)
    332             return normal_handler.handle(line_info)
    333 
--> 334         prefiltered = self.prefilter_line_info(line_info)
    335         # print "prefiltered line: %r" % prefiltered

    336         return prefiltered

/home/wesm/code/repos/ipython/IPython/core/prefilter.pyc in prefilter_line_info(self, line_info)
    272         # print "prefilter_line_info: ", line_info

    273         handler = self.find_handler(line_info)
--> 274         return handler.handle(line_info)
    275 
    276     def find_handler(self, line_info):

/home/wesm/code/repos/ipython/IPython/core/prefilter.pyc in handle(self, line_info)
    801         esc     = line_info.esc
    802         continue_prompt = line_info.continue_prompt
--> 803         obj = line_info.ofind(self)['obj']
    804         #print 'pre <%s> ifun <%s> rest <%s>' % (pre,ifun,the_rest)  # dbg

    805 

KeyError: 'obj'
```



Discussion:
Yup, confirmed.  Swamped right now, but we definitely need this one fixed, along with an added test... Thanks for the report!

The line that is triggering the problem is the `,1,2,3` line. The initial `,` is IPython syntax for applying autocalling. `,f x y` gets translated to `f('x','y')`. I'm not entirely sure why `store_or_execute()` is trying to transform that line independently. Typing that code at that IPython prompt manually does not cause any problem. Executing `,1,2,3` by itself at the command line "correctly" transforms it to `1(",2,3")` and fails as expected.

Normally we only run prefilter when the input is a single line. Is there a good reason we use it on multiline code for %paste?

On Thu, Jan 12, 2012 at 1:53 PM, Thomas
reply@reply.github.com
wrote:

> Normally we only run prefilter when the input is a single line. Is there a good reason we use it on multiline code for %paste?

Well, we still want magics and such to work in pasted blocks.

> Well, we still want magics and such to work in pasted blocks.

Of course. But we could use inputsplitter, as we do for multiline cells. It just means automagics (`paste` without the %) don't work.

On Thu, Jan 12, 2012 at 4:14 PM, Thomas
reply@reply.github.com
wrote:

> Of course. But we could use inputsplitter, as we do for multiline cells. It just means automagics (`paste` without the %) don't work.

Ah, we most certainly should be doing that!  It's probably then that
this code never got updated.  It's OK for unqualified magics to not
work here, just like they don't work in multiline cells either.

For that matter, we should probably unify inputsplitter and prefilter, as they now do largely equivalent things (since inputsplitter is no longer responsible for actually splitting input).

On Thu, Jan 12, 2012 at 5:18 PM, Thomas
reply@reply.github.com
wrote:

> For that matter, we should probably unify inputsplitter and prefilter, as they now do largely equivalent things (since inputsplitter is no longer responsible for actually splitting input).

Yes, a cleanup on this front would be great.

It's important that we maintain an entry point for users to provide
their own custom parsing of user input, as this is used in practice by
projects such as Sage.  But a simplification of this machinery would
be most welcome.

@takluyver, do you want to tackle this one?  I'm kind of triaging what's realistic for 0.13, and this one is indeed a bit of a nasty bug, but there's only so much we'll be able to chew before release...

I'll have a look, but it may be that it needs to wait for a proper refactoring of prefilter & inputsplitter, which I guess would be post 0.13.

OK, thanks.  I'll leave it to you then: if you decide not to tackle it soon, then go ahead and remove the milestone marker on it so that it doesn't appear for 0.13.  It would be great to have this cleaned up, bug given how going deep into that machinery is potentially very disruptive, the wiser course of action may just be to wait so we don't break something critical just prior to release.

Another case where this can bite you is the % magic syntax:

```
In [3]: cpaste
Pasting code; enter '--' alone on the line to stop or use Ctrl-D.
:    print('My favorite number is %d'
:          % 5)
:
:<EOF>
  File "<ipython-input-3-d00b86fe925c>", line 2
    get_ipython().magic(u'5 )')
```


Resolving patch:
diff --git a/IPython/frontend/terminal/interactiveshell.py b/IPython/frontend/terminal/interactiveshell.py
--- a/IPython/frontend/terminal/interactiveshell.py
+++ b/IPython/frontend/terminal/interactiveshell.py
@@ -29,6 +29,7 @@
 
 from IPython.core.error import TryNext, UsageError
 from IPython.core.usage import interactive_usage, default_banner
+from IPython.core.inputsplitter import IPythonInputSplitter
 from IPython.core.interactiveshell import InteractiveShell, InteractiveShellABC
 from IPython.core.magic import Magics, magics_class, line_magic
 from IPython.testing.skipdoctest import skip_doctest
@@ -37,7 +38,7 @@
 from IPython.utils.terminal import toggle_set_term_title, set_term_title
 from IPython.utils.process import abbrev_cwd
 from IPython.utils.warn import warn, error
-from IPython.utils.text import num_ini_spaces, SList
+from IPython.utils.text import num_ini_spaces, SList, strip_email_quotes
 from IPython.utils.traitlets import Integer, CBool, Unicode
 
 #-----------------------------------------------------------------------------
@@ -72,61 +73,105 @@ def get_pasted_lines(sentinel, l_input=py3compat.input):
             return
 
 
-def strip_email_quotes(raw_lines):
-    """ Strip email quotation marks at the beginning of each line.
-
-    We don't do any more input transofrmations here because the main shell's
-    prefiltering handles other cases.
-    """
-    lines = [re.sub(r'^\s*(\s?>)+', '', l) for l in raw_lines]
-    return '\n'.join(lines) + '\n'
-
+#------------------------------------------------------------------------
+# Terminal-specific magics
+#------------------------------------------------------------------------
 
-# These two functions are needed by the %paste/%cpaste magics.  In practice
-# they are basically methods (they take the shell as their first argument), but
-# we leave them as standalone functions because eventually the magics
-# themselves will become separate objects altogether.  At that point, the
-# magics will have access to the shell object, and these functions can be made
-# methods of the magic object, but not of the shell.
+@magics_class
+class TerminalMagics(Magics):
+    def __init__(self, shell):
+        super(TerminalMagics, self).__init__(shell)
+        self.input_splitter = IPythonInputSplitter(input_mode='line')
 
-def store_or_execute(shell, block, name):
-    """ Execute a block, or store it in a variable, per the user's request.
-    """
-    # Dedent and prefilter so what we store matches what is executed by
-    # run_cell.
-    b = shell.prefilter(textwrap.dedent(block))
+    def cleanup_input(self, block):
+        """Apply all possible IPython cleanups to an input block.
 
-    if name:
-        # If storing it for further editing, run the prefilter on it
-        shell.user_ns[name] = SList(b.splitlines())
-        print "Block assigned to '%s'" % name
-    else:
-        shell.user_ns['pasted_block'] = b
-        shell.run_cell(b)
+        This means:
 
+        - remove any global leading whitespace (dedent)
+        - remove any email quotes ('>') if they are present in *all* lines
+        - apply all static inputsplitter transforms and break into sub-blocks
+        - apply prefilter() to each sub-block that is a single line.
 
-def rerun_pasted(shell, name='pasted_block'):
-    """ Rerun a previously pasted command.
-    """
-    b = shell.user_ns.get(name)
+        Parameters
+        ----------
+        block : str
+          A possibly multiline input string of code.
 
-    # Sanity checks
-    if b is None:
-        raise UsageError('No previous pasted block available')
-    if not isinstance(b, basestring):
-        raise UsageError(
-            "Variable 'pasted_block' is not a string, can't execute")
+        Returns
+        -------
+        transformed block : str
+          The input, with all transformations above applied.
+        """
+        # We have to effectively implement client-side the loop that is done by
+        # the terminal frontend, and furthermore do it on a block that can
+        # possibly contain multiple statments pasted in one go.
+
+        # First, run the input through the block splitting code.  We should
+        # eventually make this a self-contained method in the inputsplitter.
+        isp = self.input_splitter
+        isp.reset()
+        b = textwrap.dedent(block)
+
+        # Remove email quotes first.  These must be consistently applied to
+        # *all* lines to be removed
+        b = strip_email_quotes(b)
+
+        # Split the input into independent sub-blocks so we can later do
+        # prefiltering (which must be done *only* to single-line inputs)
+        blocks = []
+        last_block = []
+        for line in b.splitlines():
+            isp.push(line)
+            last_block.append(line)
+            if not isp.push_accepts_more():
+                blocks.append(isp.source_reset())
+                last_block = []
+        if last_block:
+            blocks.append('\n'.join(last_block))
+
+        # Now, apply prefiltering to any one-line block to match the behavior
+        # of the interactive terminal
+        final_blocks = []
+        for block in blocks:
+            lines = block.splitlines()
+            if len(lines) == 1:
+                final_blocks.append(self.shell.prefilter(lines[0]))
+            else:
+                final_blocks.append(block)
 
-    print "Re-executing '%s...' (%d chars)"% (b.split('\n',1)[0], len(b))
-    shell.run_cell(b)
+        # We now have the final version of the input code as a list of blocks,
+        # with all inputsplitter transformations applied and single-line blocks
+        # run through prefilter.  For further processing, turn into a single
+        # string as the rest of our apis use string inputs.
+        return '\n'.join(final_blocks)
+        
+    def store_or_execute(self, block, name):
+        """ Execute a block, or store it in a variable, per the user's request.
+        """
+        b = self.cleanup_input(block)
+        if name:
+            # If storing it for further editing
+            self.shell.user_ns[name] = SList(b.splitlines())
+            print "Block assigned to '%s'" % name
+        else:
+            self.shell.user_ns['pasted_block'] = b
+            self.shell.run_cell(b)
 
+    def rerun_pasted(self, name='pasted_block'):
+        """ Rerun a previously pasted command.
+        """
+        b = self.shell.user_ns.get(name)
 
-#------------------------------------------------------------------------
-# Terminal-specific magics
-#------------------------------------------------------------------------
+        # Sanity checks
+        if b is None:
+            raise UsageError('No previous pasted block available')
+        if not isinstance(b, basestring):
+            raise UsageError(
+                "Variable 'pasted_block' is not a string, can't execute")
 
-@magics_class
-class TerminalMagics(Magics):
+        print "Re-executing '%s...' (%d chars)"% (b.split('\n',1)[0], len(b))
+        self.shell.run_cell(b)
 
     @line_magic
     def autoindent(self, parameter_s = ''):
@@ -178,15 +223,14 @@ def cpaste(self, parameter_s=''):
           :--
           Hello world!
         """
-
         opts, name = self.parse_options(parameter_s, 'rs:', mode='string')
         if 'r' in opts:
-            rerun_pasted(self.shell)
+            self.rerun_pasted()
             return
 
         sentinel = opts.get('s', '--')
-        block = strip_email_quotes(get_pasted_lines(sentinel))
-        store_or_execute(self.shell, block, name)
+        block = '\n'.join(get_pasted_lines(sentinel))
+        self.store_or_execute(block, name)
 
     @line_magic
     def paste(self, parameter_s=''):
@@ -205,7 +249,7 @@ def paste(self, parameter_s=''):
 
         You can also pass a variable name as an argument, e.g. '%paste foo'.
         This assigns the pasted block to variable 'foo' as string, without
-        dedenting or executing it (preceding >>> and + is still stripped)
+        executing it (preceding >>> and + is still stripped).
 
         Options
         -------
@@ -222,11 +266,10 @@ def paste(self, parameter_s=''):
         """
         opts, name = self.parse_options(parameter_s, 'rq', mode='string')
         if 'r' in opts:
-            rerun_pasted(self.shell)
+            self.rerun_pasted()
             return
         try:
-            text = self.shell.hooks.clipboard_get()
-            block = strip_email_quotes(text.splitlines())
+            block = self.shell.hooks.clipboard_get()
         except TryNext as clipboard_exc:
             message = getattr(clipboard_exc, 'args')
             if message:
@@ -243,7 +286,7 @@ def paste(self, parameter_s=''):
                 write('\n')
             write("## -- End pasted text --\n")
 
-        store_or_execute(self.shell, block, name)
+        self.store_or_execute(block, name)
 
     # Class-level: add a '%cls' magic only on Windows
     if sys.platform == 'win32':
diff --git a/IPython/utils/text.py b/IPython/utils/text.py
--- a/IPython/utils/text.py
+++ b/IPython/utils/text.py
@@ -488,6 +488,7 @@ def format_screen(strng):
     strng = par_re.sub('',strng)
     return strng
 
+
 def dedent(text):
     """Equivalent of textwrap.dedent that ignores unindented first line.
 
@@ -514,6 +515,7 @@ def dedent(text):
     rest = textwrap.dedent(rest)
     return '\n'.join([first, rest])
 
+
 def wrap_paragraphs(text, ncols=80):
     """Wrap multiple paragraphs to fit a specified width.
 
@@ -540,6 +542,70 @@ def wrap_paragraphs(text, ncols=80):
     return out_ps
 
 
+def long_substr(data):
+    """Return the longest common substring in a list of strings.
+    
+    Credit: http://stackoverflow.com/questions/2892931/longest-common-substring-from-more-than-two-strings-python
+    """
+    substr = ''
+    if len(data) > 1 and len(data[0]) > 0:
+        for i in range(len(data[0])):
+            for j in range(len(data[0])-i+1):
+                if j > len(substr) and all(data[0][i:i+j] in x for x in data):
+                    substr = data[0][i:i+j]
+    else:
+        substr = data[0]
+    return substr
+
+
+def strip_email_quotes(text):
+    """Strip leading email quotation characters ('>').
+
+    Removes any combination of leading '>' interspersed with whitespace that
+    appears *identically* in all lines of the input text.
+
+    Parameters
+    ----------
+    text : str
+
+    Examples
+    --------
+
+    Simple uses::
+
+        In [2]: strip_email_quotes('> > text')
+        Out[2]: 'text'
+
+        In [3]: strip_email_quotes('> > text\\n> > more')
+        Out[3]: 'text\\nmore'
+
+    Note how only the common prefix that appears in all lines is stripped::
+
+        In [4]: strip_email_quotes('> > text\\n> > more\\n> more...')
+        Out[4]: '> text\\n> more\\nmore...'
+
+    So if any line has no quote marks ('>') , then none are stripped from any
+    of them ::
+    
+        In [5]: strip_email_quotes('> > text\\n> > more\\nlast different')
+        Out[5]: '> > text\\n> > more\\nlast different'
+    """
+    lines = text.splitlines()
+    matches = set()
+    for line in lines:
+        prefix = re.match(r'^(\s*>[ >]*)', line)
+        if prefix:
+            matches.add(prefix.group(1))
+        else:
+            break
+    else:
+        prefix = long_substr(list(matches))
+        if prefix:
+            strip = len(prefix)
+            text = '\n'.join([ ln[strip:] for ln in lines])
+    return text
+
+
 class EvalFormatter(Formatter):
     """A String Formatter that allows evaluation of simple expressions.
     
@@ -564,6 +630,7 @@ def get_field(self, name, args, kwargs):
         v = eval(name, kwargs)
         return v, name
 
+
 @skip_doctest_py3
 class FullEvalFormatter(Formatter):
     """A String Formatter that allows evaluation of simple expressions.
@@ -621,6 +688,7 @@ def _vformat(self, format_string, args, kwargs, used_args, recursion_depth):
 
         return u''.join(py3compat.cast_unicode(s) for s in result)
 
+
 @skip_doctest_py3
 class DollarFormatter(FullEvalFormatter):
     """Formatter allowing Itpl style $foo replacement, for names and attribute
@@ -663,11 +731,13 @@ def parse(self, fmt_string):
 #-----------------------------------------------------------------------------
 # Utils to columnize a list of string
 #-----------------------------------------------------------------------------
+
 def _chunks(l, n):
     """Yield successive n-sized chunks from l."""
     for i in xrange(0, len(l), n):
         yield l[i:i+n]
 
+
 def _find_optimal(rlist , separator_size=2 , displaywidth=80):
     """Calculate optimal info to columnize a list of string"""
     for nrow in range(1, len(rlist)+1) :
@@ -682,6 +752,7 @@ def _find_optimal(rlist , separator_size=2 , displaywidth=80):
             'columns_width' : chk
            }
 
+
 def _get_or_default(mylist, i, default=None):
     """return list item number, or default if don't exist"""
     if i >= len(mylist):
@@ -689,6 +760,7 @@ def _get_or_default(mylist, i, default=None):
     else :
         return mylist[i]
 
+
 @skip_doctest
 def compute_item_matrix(items, empty=None, *args, **kwargs) :
     """Returns a nested list, and info to columnize items
@@ -746,6 +818,7 @@ def compute_item_matrix(items, empty=None, *args, **kwargs) :
     nrow, ncol = info['rows_numbers'], info['columns_numbers']
     return ([[ _get_or_default(items, c*nrow+i, default=empty) for c in range(ncol) ] for i in range(nrow) ], info)
 
+
 def columnize(items, separator='  ', displaywidth=80):
     """ Transform a list of strings into a single string with columns.
 


Test patch:
diff --git a/IPython/core/tests/test_magic_terminal.py b/IPython/core/tests/test_magic_terminal.py
--- a/IPython/core/tests/test_magic_terminal.py
+++ b/IPython/core/tests/test_magic_terminal.py
@@ -10,11 +10,17 @@
 
 import sys
 from StringIO import StringIO
+from unittest import TestCase
 
 import nose.tools as nt
 
 from IPython.testing import tools as tt
 
+#-----------------------------------------------------------------------------
+# Globals
+#-----------------------------------------------------------------------------
+ip = get_ipython()
+
 #-----------------------------------------------------------------------------
 # Test functions begin
 #-----------------------------------------------------------------------------
@@ -23,13 +29,12 @@ def check_cpaste(code, should_fail=False):
     """Execute code via 'cpaste' and ensure it was executed, unless
     should_fail is set.
     """
-    _ip.user_ns['code_ran'] = False
+    ip.user_ns['code_ran'] = False
 
     src = StringIO()
     if not hasattr(src, 'encoding'):
         # IPython expects stdin to have an encoding attribute
         src.encoding = None
-    src.write('\n')
     src.write(code)
     src.write('\n--\n')
     src.seek(0)
@@ -40,10 +45,10 @@ def check_cpaste(code, should_fail=False):
     try:
         context = tt.AssertPrints if should_fail else tt.AssertNotPrints
         with context("Traceback (most recent call last)"):
-                _ip.magic('cpaste')
+                ip.magic('cpaste')
 
         if not should_fail:
-            assert _ip.user_ns['code_ran']
+            assert ip.user_ns['code_ran']
     finally:
         sys.stdin = stdin_save
 
@@ -52,30 +57,30 @@ def check_cpaste(code, should_fail=False):
 def test_cpaste():
     """Test cpaste magic"""
 
-    def run():
+    def runf():
         """Marker function: sets a flag when executed.
         """
-        _ip.user_ns['code_ran'] = True
-        return 'run' # return string so '+ run()' doesn't result in success
-
-    tests = {'pass': ["run()",
-                      "In [1]: run()",
-                      "In [1]: if 1:\n   ...:     run()",
-                      "> > > run()",
-                      ">>> run()",
-                      "   >>> run()",
+        ip.user_ns['code_ran'] = True
+        return 'runf' # return string so '+ runf()' doesn't result in success
+
+    tests = {'pass': ["runf()",
+                      "In [1]: runf()",
+                      "In [1]: if 1:\n   ...:     runf()",
+                      "> > > runf()",
+                      ">>> runf()",
+                      "   >>> runf()",
                       ],
 
-             'fail': ["1 + run()",
+             'fail': ["1 + runf()",
              ]}
     
     # I don't know why this is failing specifically on Python 3.1. I've
     # checked it manually interactively, but we don't care enough about 3.1
     # to spend time fiddling with the tests, so we just skip it.
     if not PY31:
-        tests['fail'].append("++ run()")
+        tests['fail'].append("++ runf()")
 
-    _ip.user_ns['run'] = run
+    ip.user_ns['runf'] = runf
 
     for code in tests['pass']:
         check_cpaste(code)
@@ -84,87 +89,116 @@ def run():
         check_cpaste(code, should_fail=True)
 
 
-# Multiple tests for clipboard pasting
-def test_paste():
-    _ip = get_ipython()
+class PasteTestCase(TestCase):
+    """Multiple tests for clipboard pasting"""
 
-    def paste(txt, flags='-q'):
+    def paste(self, txt, flags='-q'):
         """Paste input text, by default in quiet mode"""
-        hooks.clipboard_get = lambda : txt
-        _ip.magic('paste '+flags)
+        ip.hooks.clipboard_get = lambda : txt
+        ip.magic('paste '+flags)
 
-    # Inject fake clipboard hook but save original so we can restore it later
-    hooks = _ip.hooks
-    user_ns = _ip.user_ns
-    original_clip = hooks.clipboard_get
+    def setUp(self):
+        # Inject fake clipboard hook but save original so we can restore it later
+        self.original_clip = ip.hooks.clipboard_get
 
-    try:
-        # Run tests with fake clipboard function
-        user_ns.pop('x', None)
-        paste('x=1')
-        nt.assert_equal(user_ns['x'], 1)
-
-        user_ns.pop('x', None)
-        paste('>>> x=2')
-        nt.assert_equal(user_ns['x'], 2)
-
-        paste("""
+    def tearDown(self): 
+        # Restore original hook
+        ip.hooks.clipboard_get = self.original_clip
+       
+    def test_paste(self):
+        ip.user_ns.pop('x', None)
+        self.paste('x = 1')
+        nt.assert_equal(ip.user_ns['x'], 1)
+        ip.user_ns.pop('x')
+
+    def test_paste_pyprompt(self):
+        ip.user_ns.pop('x', None)
+        self.paste('>>> x=2')
+        nt.assert_equal(ip.user_ns['x'], 2)
+        ip.user_ns.pop('x')
+
+    def test_paste_py_multi(self):
+        self.paste("""
         >>> x = [1,2,3]
         >>> y = []
         >>> for i in x:
         ...     y.append(i**2)
-        ...
+        ... 
         """)
-        nt.assert_equal(user_ns['x'], [1,2,3])
-        nt.assert_equal(user_ns['y'], [1,4,9])
-
-        # Now, test that paste -r works
-        user_ns.pop('x', None)
-        nt.assert_false('x' in user_ns)
-        _ip.magic('paste -r')
-        nt.assert_equal(user_ns['x'], [1,2,3])
-
-        # Test pasting of email-quoted contents
-        paste("""
+        nt.assert_equal(ip.user_ns['x'], [1,2,3])
+        nt.assert_equal(ip.user_ns['y'], [1,4,9])
+
+    def test_paste_py_multi_r(self):
+        "Now, test that self.paste -r works"
+        nt.assert_equal(ip.user_ns.pop('x'), [1,2,3])
+        nt.assert_equal(ip.user_ns.pop('y'), [1,4,9])
+        nt.assert_false('x' in ip.user_ns)
+        ip.magic('paste -r')
+        nt.assert_equal(ip.user_ns['x'], [1,2,3])
+        nt.assert_equal(ip.user_ns['y'], [1,4,9])
+
+    def test_paste_email(self):
+        "Test pasting of email-quoted contents"
+        self.paste("""\
         >> def foo(x):
         >>     return x + 1
-        >> x = foo(1.1)
-        """)
-        nt.assert_equal(user_ns['x'], 2.1)
+        >> xx = foo(1.1)""")
+        nt.assert_equal(ip.user_ns['xx'], 2.1)
 
-        # Email again; some programs add a space also at each quoting level
-        paste("""
+    def test_paste_email2(self):
+        "Email again; some programs add a space also at each quoting level"
+        self.paste("""\
         > > def foo(x):
         > >     return x + 1
-        > > x = foo(2.1)
-        """)
-        nt.assert_equal(user_ns['x'], 3.1)
+        > > yy = foo(2.1)     """)
+        nt.assert_equal(ip.user_ns['yy'], 3.1)
 
-        # Email quoting of interactive input
-        paste("""
+    def test_paste_email_py(self):
+        "Email quoting of interactive input"
+        self.paste("""\
         >> >>> def f(x):
         >> ...   return x+1
-        >> ...
-        >> >>> x = f(2.5)
-        """)
-        nt.assert_equal(user_ns['x'], 3.5)
+        >> ... 
+        >> >>> zz = f(2.5)      """)
+        nt.assert_equal(ip.user_ns['zz'], 3.5)
 
-        # Also test paste echoing, by temporarily faking the writer
+    def test_paste_echo(self):
+        "Also test self.paste echoing, by temporarily faking the writer"
         w = StringIO()
-        writer = _ip.write
-        _ip.write = w.write
+        writer = ip.write
+        ip.write = w.write
         code = """
         a = 100
         b = 200"""
         try:
-            paste(code,'')
+            self.paste(code,'')
             out = w.getvalue()
         finally:
-            _ip.write = writer
-        nt.assert_equal(user_ns['a'], 100)
-        nt.assert_equal(user_ns['b'], 200)
+            ip.write = writer
+        nt.assert_equal(ip.user_ns['a'], 100)
+        nt.assert_equal(ip.user_ns['b'], 200)
         nt.assert_equal(out, code+"\n## -- End pasted text --\n")
 
-    finally:
-        # Restore original hook
-        hooks.clipboard_get = original_clip
+    def test_paste_leading_commas(self):
+        "Test multiline strings with leading commas"
+        tm = ip.magics_manager.registry['TerminalMagics']
+        s = '''\
+a = """
+,1,2,3
+"""'''
+        ip.user_ns.pop('foo', None)
+        tm.store_or_execute(s, 'foo')
+        nt.assert_in('foo', ip.user_ns)
+
+
+    def test_paste_trailing_question(self):
+        "Test pasting sources with trailing question marks"
+        tm = ip.magics_manager.registry['TerminalMagics']
+        s = '''\
+def funcfoo():
+   if True: #am i true?
+       return 'fooresult'
+'''
+        ip.user_ns.pop('funcfoo', None)
+        self.paste(s)
+        nt.assert_equals(ip.user_ns['funcfoo'](), 'fooresult')
diff --git a/IPython/utils/tests/test_text.py b/IPython/utils/tests/test_text.py
--- a/IPython/utils/tests/test_text.py
+++ b/IPython/utils/tests/test_text.py
@@ -138,3 +138,33 @@ def test_dollar_formatter():
     nt.assert_equals(s, "12 $HOME")
     s = f.format("${foo}", foo="HOME")
     nt.assert_equals(s, "$HOME")
+
+
+def test_long_substr():
+    data = ['hi']
+    nt.assert_equals(text.long_substr(data), 'hi')
+
+
+def test_long_substr2():
+    data = ['abc', 'abd', 'abf', 'ab']
+    nt.assert_equals(text.long_substr(data), 'ab')
+
+
+def test_strip_email():
+    src = """\
+        >> >>> def f(x):
+        >> ...   return x+1
+        >> ... 
+        >> >>> zz = f(2.5)"""
+    cln = """\
+>>> def f(x):
+...   return x+1
+... 
+>>> zz = f(2.5)"""
+    nt.assert_equals(text.strip_email_quotes(src), cln)
+
+
+def test_strip_email2():
+    src = '> > > list()'
+    cln = 'list()'
+    nt.assert_equals(text.strip_email_quotes(src), cln)