Repository: dask/dask
Pull request: Change doctests to Python 3 #3690
This fixes #3690 

Modified travis.yml and documentation accordingly.
Modified some doctests in order to run with Python 3.6 (mainly sets, bytes, unicode, etc)

- [x] Tests added / passed
- [x] Passes `flake8 dask`


Problem statement:
Change doctests to Python 3
Subtle differences between Python 2 and 3 mean that it is inconvenient to test doctests in both versions.  Today our CI systems only run doctests in Python 2.  This policy was made several years ago, when Python 2 was more common.  

It would be nice to switch our CI to test docstrings in Python 3 instead.  This requires two changes:

1. Change the `.travis.yml` file to set `coverage` in the newest Python build (3.6 today) rather than the 2.7 build.  This coverage flag controls whether or not we run doctests (see continuous_integration/travis/run_tests.sh for more information)
2.  Fix all of the failing tests in our docstrings that used to rely on Python 2 syntax

Hints from issue discussion:

Commit message: Modified doctests in order to run with Python 3.6 #3690

Patch to review:
diff --git a/.travis.yml b/.travis.yml
index fca0b8dc03f..15cd1e5eaf9 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -20,7 +20,7 @@ jobs:
       - NUMPY=1.14.1
       - PANDAS=0.22.0
       - *test_and_lint
-      - *coverage
+      - *no_coverage
       - *no_optimize
       - *no_imports
 
@@ -57,7 +57,7 @@ jobs:
       - NUMPY=1.14.1
       - PANDAS=0.22.0
       - *test_and_lint
-      - *no_coverage
+      - *coverage
       - *no_optimize
       - *imports
 
diff --git a/dask/array/slicing.py b/dask/array/slicing.py
index 8493568c61d..98d29b223a4 100644
--- a/dask/array/slicing.py
+++ b/dask/array/slicing.py
@@ -42,7 +42,7 @@ def sanitize_index(ind):
     >>> sanitize_index(np.array([False, True, True]))
     array([1, 2])
     >>> type(sanitize_index(np.int32(0)))
-    <type 'int'>
+    <class 'int'>
     >>> sanitize_index(1.0)
     1
     >>> sanitize_index(0.5)
@@ -364,11 +364,15 @@ def _slice_1d(dim_shape, lengths, index):
 
     And negative slicing
 
-    >>> _slice_1d(100, [20, 20, 20, 20, 20], slice(100, 0, -3))
-    {0: slice(-2, -20, -3), 1: slice(-1, -21, -3), 2: slice(-3, -21, -3), 3: slice(-2, -21, -3), 4: slice(-1, -21, -3)}
+    >>> exp_res = {0: slice(-2, -20, -3), 1: slice(-1, -21, -3), 2: slice(-3, -21, -3),
+    ...            3: slice(-2, -21, -3), 4: slice(-1, -21, -3)}
+    >>> _slice_1d(100, [20, 20, 20, 20, 20], slice(100, 0, -3)) == exp_res
+    True
 
-    >>> _slice_1d(100, [20, 20, 20, 20, 20], slice(100, 12, -3))
-    {0: slice(-2, -8, -3), 1: slice(-1, -21, -3), 2: slice(-3, -21, -3), 3: slice(-2, -21, -3), 4: slice(-1, -21, -3)}
+    >>> exp_res = {0: slice(-2, -8, -3), 1: slice(-1, -21, -3), 2: slice(-3, -21, -3),
+    ...            3: slice(-2, -21, -3), 4: slice(-1, -21, -3)}
+    >>> _slice_1d(100, [20, 20, 20, 20, 20], slice(100, 12, -3)) == exp_res
+    True
 
     >>> _slice_1d(100, [20, 20, 20, 20, 20], slice(100, -12, -3))
     {4: slice(-1, -12, -3)}
diff --git a/dask/bag/core.py b/dask/bag/core.py
index 1908b80b0df..76c58e041aa 100644
--- a/dask/bag/core.py
+++ b/dask/bag/core.py
@@ -576,9 +576,9 @@ def random_sample(self, prob, random_state=None):
         >>> import dask.bag as db
         >>> b = db.from_sequence(range(5))
         >>> list(b.random_sample(0.5, 42))
-        [1, 4]
+        [1, 3]
         >>> list(b.random_sample(0.5, 42))
-        [1, 4]
+        [1, 3]
         """
         if not 0 <= prob <= 1:
             raise ValueError('prob must be a number in the interval [0, 1]')
diff --git a/dask/core.py b/dask/core.py
index 5cc3b8281bf..d25f967b57e 100644
--- a/dask/core.py
+++ b/dask/core.py
@@ -168,22 +168,22 @@ def get_dependencies(dsk, key=None, task=None, as_list=False):
     ...        'a': (add, (inc, 'x'), 1)}
 
     >>> get_dependencies(dsk, 'x')
-    set([])
+    set()
 
     >>> get_dependencies(dsk, 'y')
-    set(['x'])
+    {'x'}
 
     >>> get_dependencies(dsk, 'z')  # doctest: +SKIP
-    set(['x', 'y'])
+    {'x', 'y'}
 
     >>> get_dependencies(dsk, 'w')  # Only direct dependencies
-    set(['z'])
+    {'z'}
 
     >>> get_dependencies(dsk, 'a')  # Ignore non-keys
-    set(['x'])
+    {'x'}
 
     >>> get_dependencies(dsk, task=(inc, 'x'))  # provide tasks directly
-    set(['x'])
+    {'x'}
     """
     if key is not None:
         arg = dsk[key]
@@ -222,9 +222,9 @@ def get_deps(dsk):
     >>> dsk = {'a': 1, 'b': (inc, 'a'), 'c': (inc, 'b')}
     >>> dependencies, dependents = get_deps(dsk)
     >>> dependencies
-    {'a': set([]), 'c': set(['b']), 'b': set(['a'])}
+    {'a': set(), 'b': {'a'}, 'c': {'b'}}
     >>> dependents
-    {'a': set(['b']), 'c': set([]), 'b': set(['c'])}
+    {'a': {'b'}, 'b': {'c'}, 'c': set()}
     """
     dependencies = {k: get_dependencies(dsk, task=v)
                     for k, v in dsk.items()}
diff --git a/dask/local.py b/dask/local.py
index 6dd8eab8483..8195487fbc9 100644
--- a/dask/local.py
+++ b/dask/local.py
@@ -65,22 +65,14 @@
 >>> dsk = {'x': 1, 'y': 2, 'z': (inc, 'x'), 'w': (add, 'z', 'y')}
 >>> pprint.pprint(start_state_from_dask(dsk)) # doctest: +NORMALIZE_WHITESPACE
 {'cache': {'x': 1, 'y': 2},
- 'dependencies': {'w': set(['y', 'z']),
-                  'x': set([]),
-                  'y': set([]),
-                  'z': set(['x'])},
- 'dependents': {'w': set([]),
-                'x': set(['z']),
-                'y': set(['w']),
-                'z': set(['w'])},
- 'finished': set([]),
+ 'dependencies': {'w': {'z', 'y'}, 'x': set(), 'y': set(), 'z': {'x'}},
+ 'dependents': {'w': set(), 'x': {'z'}, 'y': {'w'}, 'z': {'w'}},
+ 'finished': set(),
  'ready': ['z'],
- 'released': set([]),
- 'running': set([]),
- 'waiting': {'w': set(['z'])},
- 'waiting_data': {'x': set(['z']),
-                  'y': set(['w']),
-                  'z': set(['w'])}}
+ 'released': set(),
+ 'running': set(),
+ 'waiting': {'w': {'z'}},
+ 'waiting_data': {'x': {'z'}, 'y': {'w'}, 'z': {'w'}}}
 
 Optimizations
 =============
@@ -164,22 +156,14 @@ def start_state_from_dask(dsk, cache=None, sortkey=None):
     >>> from pprint import pprint
     >>> pprint(start_state_from_dask(dsk)) # doctest: +NORMALIZE_WHITESPACE
     {'cache': {'x': 1, 'y': 2},
-     'dependencies': {'w': set(['y', 'z']),
-                      'x': set([]),
-                      'y': set([]),
-                      'z': set(['x'])},
-     'dependents': {'w': set([]),
-                    'x': set(['z']),
-                    'y': set(['w']),
-                    'z': set(['w'])},
-     'finished': set([]),
+     'dependencies': {'w': {'z', 'y'}, 'x': set(), 'y': set(), 'z': {'x'}},
+     'dependents': {'w': set(), 'x': {'z'}, 'y': {'w'}, 'z': {'w'}},
+     'finished': set(),
      'ready': ['z'],
-     'released': set([]),
-     'running': set([]),
-     'waiting': {'w': set(['z'])},
-     'waiting_data': {'x': set(['z']),
-                      'y': set(['w']),
-                      'z': set(['w'])}}
+     'released': set(),
+     'running': set(),
+     'waiting': {'w': {'z'}},
+     'waiting_data': {'x': {'z'}, 'y': {'w'}, 'z': {'w'}}}
     """
     if sortkey is None:
         sortkey = order(dsk).get
diff --git a/dask/optimization.py b/dask/optimization.py
index e6ff36ded46..bb8043fbb84 100644
--- a/dask/optimization.py
+++ b/dask/optimization.py
@@ -760,27 +760,27 @@ def fuse(dsk, keys=None, dependencies=None, ave_width=None, max_width=None,
 def key_split(s):
     """
     >>> key_split('x')
-    u'x'
+    'x'
     >>> key_split('x-1')
-    u'x'
+    'x'
     >>> key_split('x-1-2-3')
-    u'x'
+    'x'
     >>> key_split(('x-2', 1))
     'x'
     >>> key_split("('x-2', 1)")
-    u'x'
+    'x'
     >>> key_split('hello-world-1')
-    u'hello-world'
+    'hello-world'
     >>> key_split(b'hello-world-1')
-    u'hello-world'
+    'hello-world'
     >>> key_split('ae05086432ca935f6eba409a8ecd4896')
     'data'
     >>> key_split('<module.submodule.myclass object at 0xdaf372')
-    u'myclass'
+    'myclass'
     >>> key_split(None)
     'Other'
     >>> key_split('x-abcdefab')  # ignores hex
-    u'x'
+    'x'
     """
     if type(s) is bytes:
         s = s.decode()
diff --git a/dask/tests/test_core.py b/dask/tests/test_core.py
index b8999f3f9d3..8789d0bfe33 100644
--- a/dask/tests/test_core.py
+++ b/dask/tests/test_core.py
@@ -134,9 +134,9 @@ def test_get_deps():
     >>> dsk = {'a': 1, 'b': (inc, 'a'), 'c': (inc, 'b')}
     >>> dependencies, dependents = get_deps(dsk)
     >>> dependencies
-    {'a': set([]), 'c': set(['b']), 'b': set(['a'])}
+    {'a': set(), 'b': {'a'}, 'c': {'b'}}
     >>> dependents
-    {'a': set(['b']), 'c': set([]), 'b': set(['c'])}
+    {'a': {'b'}, 'b': {'c'}, 'c': set()}
     """
     dsk = {'a': [1, 2, 3],
            'b': 'a',
diff --git a/dask/utils.py b/dask/utils.py
index aea4390a6ba..da6f6fa0ecf 100644
--- a/dask/utils.py
+++ b/dask/utils.py
@@ -553,11 +553,11 @@ def ensure_bytes(s):
     """ Turn string or bytes to bytes
 
     >>> ensure_bytes(u'123')
-    '123'
+    b'123'
     >>> ensure_bytes('123')
-    '123'
+    b'123'
     >>> ensure_bytes(b'123')
-    '123'
+    b'123'
     """
     if isinstance(s, bytes):
         return s
@@ -571,11 +571,11 @@ def ensure_unicode(s):
     """ Turn string or bytes to bytes
 
     >>> ensure_unicode(u'123')
-    u'123'
+    '123'
     >>> ensure_unicode('123')
-    u'123'
+    '123'
     >>> ensure_unicode(b'123')
-    u'123'
+    '123'
     """
     if isinstance(s, unicode):
         return s
diff --git a/docs/source/develop.rst b/docs/source/develop.rst
index ecf9b83a6f8..1811e96639b 100644
--- a/docs/source/develop.rst
+++ b/docs/source/develop.rst
@@ -195,7 +195,7 @@ after the line.
 
 .. _numpydoc: https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt
 
-Docstrings are currently tested under Python 2.7 on travis.ci.  You can test
+Docstrings are currently tested under Python 3.6 on travis.ci.  You can test
 docstrings with pytest as follows::
 
    py.test dask --doctest-modules


Code review comments:
diff_hunk: @@ -364,11 +364,15 @@ def _slice_1d(dim_shape, lengths, index):
 
     And negative slicing
 
-    >>> _slice_1d(100, [20, 20, 20, 20, 20], slice(100, 0, -3))
-    {0: slice(-2, -20, -3), 1: slice(-1, -21, -3), 2: slice(-3, -21, -3), 3: slice(-2, -21, -3), 4: slice(-1, -21, -3)}
+    >>> exp_res = {0: slice(-2, -20, -3), 1: slice(-1, -21, -3), 2: slice(-3, -21, -3),
+    ...            3: slice(-2, -21, -3), 4: slice(-1, -21, -3)}
+    >>> _slice_1d(100, [20, 20, 20, 20, 20], slice(100, 0, -3)) == exp_res
+    True
 
-    >>> _slice_1d(100, [20, 20, 20, 20, 20], slice(100, 12, -3))
-    {0: slice(-2, -8, -3), 1: slice(-1, -21, -3), 2: slice(-3, -21, -3), 3: slice(-2, -21, -3), 4: slice(-1, -21, -3)}
+    >>> exp_res = {0: slice(-2, -8, -3), 1: slice(-1, -21, -3), 2: slice(-3, -21, -3),
+    ...            3: slice(-2, -21, -3), 4: slice(-1, -21, -3)}
+    >>> _slice_1d(100, [20, 20, 20, 20, 20], slice(100, 12, -3)) == exp_res
+    True
path: dask/array/slicing.py
text: @user1:
Would it be possible to include full results here in the output rather than explicitly test?  In Python 3.6 we can probably depend on deterministic dict ordering.  We could probably also use the `# doctest: +NORMALIZE_WHITESPACE` pragma to help improve the styling, perhaps something like the following?

```python

>>> _slice_1d(100, [20, 20, 20, 20, 20], slice(100, 12, -3))  # doctest: +NORMALIZE_WHITESPACE
{0: slice(-2, -8, -3), 
 1: slice(-1, -21, -3), 
 2: slice(-3, -21, -3),
 3: slice(-2, -21, -3), 
 4: slice(-1, -21, -3)}
￼

@author:
I originally used the `# doctest: +NORMALIZE_WHITESPACE` pragma, but in the end I stuck to the doctests documentation workaround: https://docs.python.org/3.6/library/doctest.html#warnings

I agree that with full results and pragma we can improve readability and styling, I will have a look and I will modify this part.