Repository: ipython/ipython

Issue:
Pretty printing PyPy dicts not as "dictproxy"
I'm using IPython pretty printer in my plugin for running doctests with a custom representation printer. It actually works fine in CPython, but it prints all PyPy dicts as proxies.

In CPython 2.7.12 a dict isn't a dictproxy, and it's well known that they appear with curly brackets:

``` python
In [1]: {"this": 1, "is": "a", "test": None}
Out[1]: {'is': 'a', 'test': None, 'this': 1}

In [2]: import types

In [3]: types.DictProxyType is dict
Out[3]: False
```

But in PyPy 5.3.1 (Python 2.7.10), a dict appears as a dictproxy, and these types are actually the same w.r.t. the types module:

``` python
In [1]: {"this": 1, "is": "a", "test": None}
Out[1]: <dictproxy {'is': 'a', 'test': None, 'this': 1}>

In [2]: import types

In [3]: types.DictProxyType is dict
Out[3]: True

In [4]: dict.__name__
Out[4]: 'dict'

In [5]: repr(_1)
Out[5]: "{'this': 1, 'is': 'a', 'test': None}"
```

The last outputs suggests that the assignment to `_type_pprinters[types.DictProxyType]` in IPython.lib.pretty replaces the dict printer. Shouldn't that assignment be done with the setdefault method instead?



Discussion:
Hi there, 

Thanks for the investigation, and the detail explanation. Sorry if my question is dumb, I'm  jetlagged and it's late. If in PyPy dict == dictproxy, we have no way to differentiate both? So I'm not sure what you want to print? 

Also we don't overwrite  `repr()`you have to try displaying with `IPython.display.display(object)` that will pick up the `_repr_pretty_`:

```
In [1]: d =  {"this": 1, "is": "a", "test": None}

In [2]: from IPython.display import display

In [3]: display(d)
<dictproxy {'is': 'a', 'test': None, 'this': 1}>
```

Nice to see SO using IPython on pypy BTW :-) 

Oh ! I think I know what you mean. 

Yes, make sens, do you want to send a PR ? 

I'll send the PR. I was trying to find a standardized way to test the code, I found one but I'm not sure if it's right. I modified tox.ini to be able to test with PyPy installing the test requirements described in setup.py, but I'm not understanding the test suite (tox.ini and .travis.yml have different ways to run them and there's no setup.py test command). I managed to run tests with tox calling `iptest --all`, but a lot of tests fail in PyPy.

``` bash
$ tox -e pypy
pypy recreate: /home/danilo/code/github-ipython/ipython/.tox/pypy
[...]
FAILED (SKIP=14, errors=41, failures=6)
[...]
FAILED (SKIP=5, errors=2, failures=4)
[...]
FAILED (SKIP=8, failures=2)
[...]
______________________________________________________________________
Test suite completed for system with the following information:
IPython version: 5.1.0.dev
IPython commit : 58a2cf4 (installation)
IPython package: ~/code/github-ipython/ipython/.tox/pypy/site-packages/IPython
Python version : 2.7.10 (7e8df3df9641, Jun 28 2016, 23:19:52)[PyPy 5.3.1 with GCC 6.1.1 20160602]
sys.executable : ~/code/github-ipython/ipython/.tox/pypy/bin/pypy
Platform       : Linux-4.6.4-1-ARCH-x86_64-with-glibc2.2.5

Tools and libraries available at test time:
   pygments sqlite3

Tools and libraries NOT available at test time:
   matplotlib

Status: ERROR - 3 out of 7 test groups failed (core, lib, utils). Took 95.694s.

You may wish to rerun these, with:
  iptest core lib utils
```

When I apply the proposed change, only this line changes:

```
FAILED (SKIP=5, errors=2, failures=4)
```

it becomes:

```
FAILED (SKIP=5, errors=2, failures=1)
```

Which means it's something that is already being tested somewhere. Should I add some dict representation printer specific test on test_pretty.py, and some other test for dictproxy (that skips on PyPy)? If so, how do I create a dictproxy?

Testing on these is ok no matter the single line change (all with matplotlib, pygments and sqlite3, but PyPy which is without matplotlib):

```
$ tox -r
[...]
Python version : 2.7.12 (default, Jun 28 2016, 08:31:05) [GCC 6.1.1 20160602]
[...]
Python version : 3.3.6 (default, Jun 22 2016, 03:31:41) [GCC 6.1.1 20160602]
[...]
Python version : 3.4.4 (default, Jun 22 2016, 03:53:42) [GCC 6.1.1 20160602]
[...]
Python version : 3.5.2 (default, Jun 28 2016, 08:46:01) [GCC 6.1.1 20160602]
[...]
Python version : 2.7.10 (7e8df3df9641, Jun 28 2016, 23:19:52)[PyPy 5.3.1 with GCC 6.1.1 20160602]
[...]
___________________________________ summary ____________________________________
  py27: commands succeeded
  py33: commands succeeded
  py34: commands succeeded
  py35: commands succeeded
ERROR:   pypy: commands failed
```

Should I add my modified tox.ini to the PR? And a modified .travis.yml (adding it to allow_failure by now)? Or just a single line PR with the proposed change?

@danilobellini to run the tests locally (anywhere, regardless of Python version or implementation), generally do:

```
pip install ipython[test]
iptest
```

The tox config is used by nobody that I'm aware of, and should perhaps be removed, as its presence gives the misleading impression that it is supported or maintained.

We haven't run CI on PyPy, so there are likely various failures due to minor issues. If you want to go through the test suite and find PyPy failures, that would be great! But adding it to allow_failures along with nightly is a sensible start.


Resolving patch:
diff --git a/IPython/lib/pretty.py b/IPython/lib/pretty.py
--- a/IPython/lib/pretty.py
+++ b/IPython/lib/pretty.py
@@ -85,7 +85,7 @@ def _repr_pretty_(self, p, cycle):
 import datetime
 from collections import deque
 
-from IPython.utils.py3compat import PY3, cast_unicode, string_types
+from IPython.utils.py3compat import PY3, PYPY, cast_unicode, string_types
 from IPython.utils.encoding import get_stream_enc
 
 from io import StringIO
@@ -605,7 +605,8 @@ def inner(obj, p, cycle):
 
         if cycle:
             return p.text('{...}')
-        p.begin_group(1, start)
+        step = len(start)
+        p.begin_group(step, start)
         keys = obj.keys()
         # if dict isn't large enough to be truncated, sort keys before displaying
         if not (p.max_seq_length and len(obj) >= p.max_seq_length):
@@ -621,7 +622,7 @@ def inner(obj, p, cycle):
             p.pretty(key)
             p.text(': ')
             p.pretty(obj[key])
-        p.end_group(1, end)
+        p.end_group(step, end)
     return inner
 
 
@@ -631,7 +632,11 @@ def _super_pprint(obj, p, cycle):
     p.pretty(obj.__thisclass__)
     p.text(',')
     p.breakable()
-    p.pretty(obj.__self__)
+    if PYPY: # In PyPy, super() objects don't have __self__ attributes
+        dself = obj.__repr__.__self__
+        p.pretty(None if dself is obj else dself)
+    else:
+        p.pretty(obj.__self__)
     p.end_group(8, '>')
 
 
@@ -665,8 +670,10 @@ def _type_pprint(obj, p, cycle):
     # Heap allocated types might not have the module attribute,
     # and others may set it to None.
 
-    # Checks for a __repr__ override in the metaclass
-    if type(obj).__repr__ is not type.__repr__:
+    # Checks for a __repr__ override in the metaclass. Can't compare the
+    # type(obj).__repr__ directly because in PyPy the representation function
+    # inherited from type isn't the same type.__repr__
+    if [m for m in _get_mro(type(obj)) if "__repr__" in vars(m)][:1] != [type]:
         _repr_pprint(obj, p, cycle)
         return
 
@@ -753,10 +760,15 @@ def _exception_pprint(obj, p, cycle):
 }
 
 try:
-    _type_pprinters[types.DictProxyType] = _dict_pprinter_factory('<dictproxy {', '}>')
+    # In PyPy, types.DictProxyType is dict, setting the dictproxy printer
+    # using dict.setdefault avoids overwritting the dict printer
+    _type_pprinters.setdefault(types.DictProxyType,
+                               _dict_pprinter_factory('dict_proxy({', '})'))
     _type_pprinters[types.ClassType] = _type_pprint
     _type_pprinters[types.SliceType] = _repr_pprint
 except AttributeError: # Python 3
+    _type_pprinters[types.MappingProxyType] = \
+        _dict_pprinter_factory('mappingproxy({', '})')
     _type_pprinters[slice] = _repr_pprint
     
 try:
diff --git a/IPython/utils/py3compat.py b/IPython/utils/py3compat.py
--- a/IPython/utils/py3compat.py
+++ b/IPython/utils/py3compat.py
@@ -6,6 +6,7 @@
 import re
 import shutil
 import types
+import platform
 
 from .encoding import DEFAULT_ENCODING
 
@@ -292,6 +293,7 @@ def execfile(fname, glob=None, loc=None, compiler=None):
 
 
 PY2 = not PY3
+PYPY = platform.python_implementation() == "PyPy"
 
 
 def annotate(**kwargs):


Test patch:
diff --git a/IPython/lib/tests/test_pretty.py b/IPython/lib/tests/test_pretty.py
--- a/IPython/lib/tests/test_pretty.py
+++ b/IPython/lib/tests/test_pretty.py
@@ -7,11 +7,13 @@
 from __future__ import print_function
 
 from collections import Counter, defaultdict, deque, OrderedDict
+import types, string, ctypes
 
 import nose.tools as nt
 
 from IPython.lib import pretty
-from IPython.testing.decorators import skip_without, py2_only
+from IPython.testing.decorators import (skip_without, py2_only, py3_only,
+                                        cpython2_only)
 from IPython.utils.py3compat import PY3, unicode_to_str
 
 if PY3:
@@ -186,12 +188,14 @@ class SB(SA):
     pass
 
 def test_super_repr():
+    # "<super: module_name.SA, None>"
     output = pretty.pretty(super(SA))
-    nt.assert_in("SA", output)
+    nt.assert_regexp_matches(output, r"<super: \S+.SA, None>")
 
+    # "<super: module_name.SA, <module_name.SB at 0x...>>"
     sb = SB()
     output = pretty.pretty(super(SA, sb))
-    nt.assert_in("SA", output)
+    nt.assert_regexp_matches(output, r"<super: \S+.SA,\s+<\S+.SB at 0x\S+>>")
 
 
 def test_long_list():
@@ -436,3 +440,97 @@ class MyCounter(Counter):
     ]
     for obj, expected in cases:
         nt.assert_equal(pretty.pretty(obj), expected)
+
+@py3_only
+def test_mappingproxy():
+    MP = types.MappingProxyType
+    underlying_dict = {}
+    mp_recursive = MP(underlying_dict)
+    underlying_dict[2] = mp_recursive
+    underlying_dict[3] = underlying_dict
+
+    cases = [
+        (MP({}), "mappingproxy({})"),
+        (MP({None: MP({})}), "mappingproxy({None: mappingproxy({})})"),
+        (MP({k: k.upper() for k in string.ascii_lowercase}),
+         "mappingproxy({'a': 'A',\n"
+         "              'b': 'B',\n"
+         "              'c': 'C',\n"
+         "              'd': 'D',\n"
+         "              'e': 'E',\n"
+         "              'f': 'F',\n"
+         "              'g': 'G',\n"
+         "              'h': 'H',\n"
+         "              'i': 'I',\n"
+         "              'j': 'J',\n"
+         "              'k': 'K',\n"
+         "              'l': 'L',\n"
+         "              'm': 'M',\n"
+         "              'n': 'N',\n"
+         "              'o': 'O',\n"
+         "              'p': 'P',\n"
+         "              'q': 'Q',\n"
+         "              'r': 'R',\n"
+         "              's': 'S',\n"
+         "              't': 'T',\n"
+         "              'u': 'U',\n"
+         "              'v': 'V',\n"
+         "              'w': 'W',\n"
+         "              'x': 'X',\n"
+         "              'y': 'Y',\n"
+         "              'z': 'Z'})"),
+        (mp_recursive, "mappingproxy({2: {...}, 3: {2: {...}, 3: {...}}})"),
+        (underlying_dict,
+         "{2: mappingproxy({2: {...}, 3: {...}}), 3: {...}}"),
+    ]
+    for obj, expected in cases:
+        nt.assert_equal(pretty.pretty(obj), expected)
+
+@cpython2_only # In PyPy, types.DictProxyType is dict
+def test_dictproxy():
+    # This is the dictproxy constructor itself from the Python API,
+    DP = ctypes.pythonapi.PyDictProxy_New
+    DP.argtypes, DP.restype = (ctypes.py_object,), ctypes.py_object
+
+    underlying_dict = {}
+    mp_recursive = DP(underlying_dict)
+    underlying_dict[0] = mp_recursive
+    underlying_dict[-3] = underlying_dict
+
+    cases = [
+        (DP({}), "dict_proxy({})"),
+        (DP({None: DP({})}), "dict_proxy({None: dict_proxy({})})"),
+        (DP({k: k.lower() for k in string.ascii_uppercase}),
+         "dict_proxy({'A': 'a',\n"
+         "            'B': 'b',\n"
+         "            'C': 'c',\n"
+         "            'D': 'd',\n"
+         "            'E': 'e',\n"
+         "            'F': 'f',\n"
+         "            'G': 'g',\n"
+         "            'H': 'h',\n"
+         "            'I': 'i',\n"
+         "            'J': 'j',\n"
+         "            'K': 'k',\n"
+         "            'L': 'l',\n"
+         "            'M': 'm',\n"
+         "            'N': 'n',\n"
+         "            'O': 'o',\n"
+         "            'P': 'p',\n"
+         "            'Q': 'q',\n"
+         "            'R': 'r',\n"
+         "            'S': 's',\n"
+         "            'T': 't',\n"
+         "            'U': 'u',\n"
+         "            'V': 'v',\n"
+         "            'W': 'w',\n"
+         "            'X': 'x',\n"
+         "            'Y': 'y',\n"
+         "            'Z': 'z'})"),
+        (mp_recursive, "dict_proxy({-3: {-3: {...}, 0: {...}}, 0: {...}})"),
+    ]
+    for obj, expected in cases:
+        nt.assert_is_instance(obj, types.DictProxyType) # Meta-test
+        nt.assert_equal(pretty.pretty(obj), expected)
+    nt.assert_equal(pretty.pretty(underlying_dict),
+                    "{-3: {...}, 0: dict_proxy({-3: {...}, 0: {...}})}")
diff --git a/IPython/testing/decorators.py b/IPython/testing/decorators.py
--- a/IPython/testing/decorators.py
+++ b/IPython/testing/decorators.py
@@ -48,7 +48,7 @@
 from IPython.external.decorators import *
 
 # For onlyif_cmd_exists decorator
-from IPython.utils.py3compat import string_types, which, PY2, PY3
+from IPython.utils.py3compat import string_types, which, PY2, PY3, PYPY
 
 #-----------------------------------------------------------------------------
 # Classes and functions
@@ -336,6 +336,7 @@ def skip_file_no_x11(name):
 known_failure_py3 = knownfailureif(sys.version_info[0] >= 3, 
                                     'This test is known to fail on Python 3.')
 
+cpython2_only = skipif(PY3 or PYPY, "This test only runs on CPython 2.")
 py2_only = skipif(PY3, "This test only runs on Python 2.")
 py3_only = skipif(PY2, "This test only runs on Python 3.")