Repository: ipython/ipython

Issue:
Test failures and docs don't build on Mac OS X Lion
I have installed the latest ipython from github (f30a572977be51a1b928788540e323ab9bc9c58f) via `python setup.py install --user`.

There are a few test failures and the docs don't build:
http://dl.dropbox.com/u/4923986/bug_reports/ipython-tests.log
http://dl.dropbox.com/u/4923986/bug_reports/ipython-docs.log

Let me know if I should create separate tickets for some of these issues. It's hard for me to tell which are known and which are new. I know that the docs v3 problem wasn't there a few weeks ago. 

Note that I am using iterm2 (http://www.iterm2.com/, Build 1.0.0.20110821) instead of the default Apple terminal, in case it matters.



Discussion:
Hm, it might matter that it's iTerm, and it might also be that you are using MacPorts Python (you are, right?).

The doc builds have always been fragile, but these test failures are new to me (I've been running the test suite pretty much daily on Lion with System Python and Terminal.app).  Can you try running from regular Terminal and/or with iTerm and regular Python to try to isolate the problem?

Yes, I was using Macports python (python27 @2.7.2_4 to be precise).

I did have another issue with ipython and iterm2 (using help with ? didn't work because TERM was set to "xterm" by default by iterm2), but I think it is unrelated to the test failures, because they also show up in Terminal.app (see below):
http://code.google.com/p/iterm2/issues/detail?id=1956

Here's the log from running the tests in Terminal.app with Macports python:
http://dl.dropbox.com/u/4923986/bug_reports/ipython-tests-terminalapp.log

And running the tests in Terminal.app with Apple Python
http://dl.dropbox.com/u/4923986/bug_reports/ipython-tests-terminalapp-systempython.log

Note that I did not re-install ipython with Apple Python, so I don't know if the second log is useful at all.

I _think_ if you do a clean reinstall of IPython, the first failure (profile lists not matching) should go away.  The `pylab` profile has been removed, and you may have a stale install (or simply build dir) where this file still exists.  If you cleanup your IPython install and build/dist dirs, I would expect this one to go away.

The other failure is clearly encoding related, and thus _might_ somehow have to do with the `xterm` setting.  All the locale stuff is very confusing, and clearly IPython is making some assumption it shouldn't be.  Have you seen any weird unicode behavior in IPython itself (printing unicode, or entering unicode literals)?  It's possible this one is just an issue in the test suite itself.

What do you get for: 

```
python -c 'import locale; print locale.getpreferredencoding()'
```

?

Also, have you installed GNU readline? If not, you'll probably get a big warning about libedit when you start ipython, suggesting you do `easy_install readline`.

I _believe_ macports Python is built with real readline.

The first failure did go away after removing old ipython files.

Now two failures remain:

```
**********************************************************************
IPython test group: IPython.frontendFAIL: Test that multiline history skips empty line cells
FAIL: Test that multiline entries are replaced twice
```

Here's my locale encoding (both in iterm2.app and Terminal.app):

``` bash
$ python -c 'import locale; print locale.getpreferredencoding()'
US-ASCII
```

Pasting unicode characters from the clipboard into the terminal or ipython doesn't work.
Actually I don't care for that functionality, but if you want to debug why it's not working, let me know how I can help.

Yes, Macports does install GNU readline:

```
$ port installed | grep readline
  py27-readline @6.2.2_0 (active)
  py32-readline @6.2.2_0 (active)
  readline @6.2.000_0 (active)
```

That's the issue right there.  It's unusual to have an ASCII locale, which makes all unicode inputs invalid, and end up getting replaced by `?`.  We should do one of:
- not use unicode in this test
- skip this test if unicode is not supported
- perform the same (cast_bytes?) transform on the test output that we do on the input, so that unicode behavior is consistent, and not assumed to work.

After updating to 825a832c5e2969f01da6c024ae356a5f066a6495 and completely removing the previously installed ipython version the sphinx docs build now completes (with plenty of errors and warnings though):
https://dl.dropbox.com/u/4923986/bug_reports/2012-06-10/ipython-docs.txt

@cdeil - does this mean we should close the issue, or are you still having problems?  (tons of errors/warnings are always expected when running sphinx, at least in my experience, I only care that it completes).

The sphinx build does finish.

The two unicode-related test failures in IPython.frontend for which you already suggest several possible solutions above are still there.


Resolving patch:
diff --git a/IPython/utils/sysinfo.py b/IPython/utils/sysinfo.py
--- a/IPython/utils/sysinfo.py
+++ b/IPython/utils/sysinfo.py
@@ -23,7 +23,7 @@
 from ConfigParser import ConfigParser
 
 from IPython.core import release
-from IPython.utils import py3compat, _sysinfo
+from IPython.utils import py3compat, _sysinfo, encoding
 
 #-----------------------------------------------------------------------------
 # Code
@@ -91,6 +91,7 @@ def pkg_info(pkg_path):
         sys_platform=sys.platform,
         platform=platform.platform(),
         os_name=os.name,
+        default_encoding=encoding.DEFAULT_ENCODING,
         )
 
 


Test patch:
diff --git a/IPython/core/tests/test_history.py b/IPython/core/tests/test_history.py
--- a/IPython/core/tests/test_history.py
+++ b/IPython/core/tests/test_history.py
@@ -93,7 +93,7 @@ def test_history():
             # Cross testing: check that magic %save can get previous session.
             testfilename = os.path.realpath(os.path.join(tmpdir, "test.py"))
             ip.magic("save " + testfilename + " ~1/1-3")
-            with py3compat.open(testfilename) as testfile:
+            with py3compat.open(testfilename, encoding='utf-8') as testfile:
                 nt.assert_equal(testfile.read(),
                                         u"# coding: utf-8\n" + u"\n".join(hist))
 
diff --git a/IPython/frontend/terminal/console/tests/test_console.py b/IPython/frontend/terminal/console/tests/test_console.py
--- a/IPython/frontend/terminal/console/tests/test_console.py
+++ b/IPython/frontend/terminal/console/tests/test_console.py
@@ -44,16 +44,16 @@ def test_console_starts():
         raise SkipTest("Could not determine ipython command")
     
     p = pexpect.spawn(ipython_cmd, args=['console', '--colors=NoColor'])
-    idx = p.expect([r'In \[\d+\]', pexpect.EOF], timeout=4)
+    idx = p.expect([r'In \[\d+\]', pexpect.EOF], timeout=15)
     nt.assert_equals(idx, 0, "expected in prompt")
     p.sendline('5')
-    idx = p.expect([r'Out\[\d+\]: 5', pexpect.EOF], timeout=1)
+    idx = p.expect([r'Out\[\d+\]: 5', pexpect.EOF], timeout=5)
     nt.assert_equals(idx, 0, "expected out prompt")
-    idx = p.expect([r'In \[\d+\]', pexpect.EOF], timeout=1)
+    idx = p.expect([r'In \[\d+\]', pexpect.EOF], timeout=5)
     nt.assert_equals(idx, 0, "expected second in prompt")
     # send ctrl-D;ctrl-D to exit
     p.sendeof()
     p.sendeof()
-    p.expect([pexpect.EOF, pexpect.TIMEOUT], timeout=1)
+    p.expect([pexpect.EOF, pexpect.TIMEOUT], timeout=5)
     if p.isalive():
         p.terminate()
diff --git a/IPython/frontend/terminal/tests/test_interactivshell.py b/IPython/frontend/terminal/tests/test_interactivshell.py
--- a/IPython/frontend/terminal/tests/test_interactivshell.py
+++ b/IPython/frontend/terminal/tests/test_interactivshell.py
@@ -19,6 +19,7 @@
 import unittest
 
 from IPython.testing.decorators import skipif
+from IPython.utils import py3compat
 
 class InteractiveShellTestCase(unittest.TestCase):
     def rl_hist_entries(self, rl, n):
@@ -124,7 +125,10 @@ def test_replace_multiline_hist_replaces_twice(self):
         self.assertEquals(ip.readline.get_current_history_length(),
                           hlen_b4_cell)
         hist = self.rl_hist_entries(ip.readline, 3)
-        self.assertEquals(hist, ['line0', 'l€ne1\nline2', 'l€ne3\nline4'])
+        expected = [u'line0', u'l€ne1\nline2', u'l€ne3\nline4']
+        # perform encoding, in case of casting due to ASCII locale
+        expected = [ py3compat.unicode_to_str(e) for e in expected ]
+        self.assertEquals(hist, expected)
 
 
     @skipif(not get_ipython().has_readline, 'no readline')
@@ -158,4 +162,7 @@ def test_replace_multiline_hist_replaces_empty_line(self):
                           hlen_b4_cell)
         hist = self.rl_hist_entries(ip.readline, 4)
         # expect no empty cells in history
-        self.assertEquals(hist, ['line0', 'l€ne1\nline2', 'l€ne3', 'line4'])
+        expected = [u'line0', u'l€ne1\nline2', u'l€ne3', u'line4']
+        # perform encoding, in case of casting due to ASCII locale
+        expected = [ py3compat.unicode_to_str(e) for e in expected ]
+        self.assertEquals(hist, expected)