Repository: dask/dask

Issue:
Slicing start value of 0 and negative step size incorrect on Dask Arrays
When trying to slice a Dask Array with a negative step size and a start value of `0`, the generated result doesn't match what one would get from a NumPy array or ordinary slicing of a Python sequence instance. Example and environment details included below.

<details>
<summary>Example:</summary>

```python
In [1]: import numpy as np

In [2]: import dask.array as da

In [3]: a = np.arange(10)

In [4]: d = da.from_array(a, chunks=3)

In [5]: a[::-1]
Out[5]: array([9, 8, 7, 6, 5, 4, 3, 2, 1, 0])

In [6]: d[::-1].compute()
Out[6]: array([9, 8, 7, 6, 5, 4, 3, 2, 1, 0])

In [7]: a[0::-1]
Out[7]: array([0])

In [8]: d[0::-1].compute()
Out[8]: array([9, 8, 7, 6, 5, 4, 3, 2, 1, 0])
```

</details>

<br>


<details>
<summary>Environment:</summary>

```yaml
name: test
channels:
  - conda-forge
  - defaults
dependencies:
  - appnope=0.1.0=py36_0
  - backcall=0.1.0=py_0
  - blas=1.1=openblas
  - bokeh=0.12.16=py36_0
  - ca-certificates=2018.4.16=0
  - certifi=2018.4.16=py36_0
  - click=6.7=py_1
  - cloudpickle=0.5.3=py_0
  - cycler=0.10.0=py36_0
  - cytoolz=0.9.0.1=py36_0
  - dask=0.17.5=py_0
  - dask-core=0.17.5=py_0
  - decorator=4.3.0=py_0
  - distributed=1.21.8=py36_0
  - freetype=2.8.1=0
  - graphviz=2.38.0=7
  - heapdict=1.0.0=py36_0
  - ipython=6.4.0=py36_0
  - ipython_genutils=0.2.0=py36_0
  - jedi=0.12.0=py36_0
  - jinja2=2.10=py36_0
  - jpeg=9b=2
  - kiwisolver=1.0.1=py36_1
  - libgfortran=3.0.0=0
  - libpng=1.6.34=0
  - libtiff=4.0.9=0
  - locket=0.2.0=py36_1
  - markupsafe=1.0=py36_0
  - matplotlib=2.2.2=py36_1
  - msgpack-python=0.5.6=py36h2d50403_2
  - ncurses=5.9=10
  - numpy=1.14.3=py36_blas_openblas_200
  - openblas=0.2.20=8
  - openssl=1.0.2o=0
  - packaging=17.1=py_0
  - pandas=0.23.0=py36_1
  - parso=0.2.1=py_0
  - partd=0.3.8=py36_0
  - pexpect=4.6.0=py36_0
  - pickleshare=0.7.4=py36_0
  - prompt_toolkit=1.0.15=py36_0
  - psutil=5.4.5=py36_0
  - ptyprocess=0.5.2=py36_0
  - pygments=2.2.0=py36_0
  - pyparsing=2.2.0=py36_0
  - python=3.6.5=1
  - python-dateutil=2.7.3=py_0
  - python-graphviz=0.8.3=py36_0
  - pytz=2018.4=py_0
  - pyyaml=3.12=py36_1
  - readline=7.0=0
  - setuptools=39.2.0=py36_0
  - simplegeneric=0.8.1=py36_0
  - six=1.11.0=py36_1
  - sortedcontainers=2.0.2=py36_0
  - sqlite=3.20.1=2
  - tblib=1.3.2=py36_0
  - tk=8.6.7=0
  - toolz=0.9.0=py_0
  - tornado=5.0.2=py36_0
  - traitlets=4.3.2=py36_0
  - wcwidth=0.1.7=py36_0
  - xz=5.2.3=0
  - yaml=0.1.7=0
  - zict=0.1.3=py_0
  - zlib=1.2.11=h470a237_3
```

</details>


Discussion:
I'm new to dask and is interested in contributing, and this issue looks like it could be a nice first contribution (definitely correct me if I'm wrong) so I did some debugging around.

The crux seem to be https://github.com/dask/dask/blob/master/dask/array/slicing.py#L739, where dask incorrectly set `start` to `None` while it should be kept as 0.

I guess a fix would involve looking through that function and make sure it encodes the logic at https://docs.python.org/3.8/library/stdtypes.html#sequence-types-list-tuple-range (specifically, item 4 and 5 in the "Notes" section of 4.6.1), and add a bunch of tests to test_slicing.py. Does that sound reasonable?
Hi @ZiyaoWei, welcome. :) Happy to have you.

Yeah that slice normalization code doesn't look quite right. FWIW [this problem is solved]( https://kenjutsu.readthedocs.io/en/latest/kenjutsu.format.html#kenjutsu.format.reformat_slice ) by an [external library]( https://github.com/jakirkham/kenjutsu ) I wrote a while back just to work with slicing. Maybe we should just require it and use that functionality?

Let's see what others think. cc @mrocklin @jcrist @shoyer
The builtin `slice.indices` method makes this really easy.

`normalize_slice` should probably just use `slice(*idx.indices(dim))` internally for this.
Wow! I did not even know that was a thing. Thanks for teaching me something new. 😄

In case others (like me) are curious what this does.

```python
In [1]: s = slice(1, 2)

In [2]: s.indices(0)
Out[2]: (0, 0, 1)

In [3]: s.indices(1)
Out[3]: (1, 1, 1)

In [4]: s.indices(2)
Out[4]: (1, 2, 1)
```
Thanks @shoyer! That's a neat function I didn't know about :-)

However, I looked into its behavior a bit and it's probably not exactly what's needed here (see this bug report that's been closed as wontfix: https://bugs.python.org/issue1256786) - specifically, it doesn't work with the test case that @jakirkham created this issue for:

````
>>> s = slice(0, None, -1)
>>> range(10)[s]
[0]
>>> range(10)[slice(*s.indices(10))]
[]
````

The root cause is indices is returning `(0, -1, -1)` and Python interprets the -1 as counting from the end. Not sure what's the best course of action here - I'll going to do some more debugging/digging once I get more time this weekend. Also not sure if I am missing something obvious :-/
Here's the mailing list discussion:
https://mail.python.org/pipermail/python-list/2005-August/312783.html

In particular:

> What the values returned from indices are for is to pass to the
range() function, more or less.  They're not intended to be
interpreted in the way things passed to __getitem__ are.

I see now that that is how we use it in xarray: https://github.com/pydata/xarray/blob/a1fa397c48698b3434e513d480a8a2e8ad04dd58/xarray/core/indexing.py#L48

But it might also work to explicitly slice a range object, and then reconstruct the slice from the result, e.g.,
```
range_obj = range(*old_slice.indices(length))
# make new_slice from range_obj
```

This is similar to what we do here in xarray: https://github.com/pydata/xarray/blob/a1fa397c48698b3434e513d480a8a2e8ad04dd58/xarray/core/indexing.py#L258

Resolving patch:
diff --git a/dask/array/slicing.py b/dask/array/slicing.py
--- a/dask/array/slicing.py
+++ b/dask/array/slicing.py
@@ -718,23 +718,21 @@ def normalize_slice(idx, dim):
     """
 
     if isinstance(idx, slice):
-        start, stop, step = idx.start, idx.stop, idx.step
-        if start is not None:
-            if start < 0 and not math.isnan(dim):
-                start = max(0, start + dim)
-            elif start > dim:
-                start = dim
-        if stop is not None:
-            if stop < 0 and not math.isnan(dim):
-                stop = max(0, stop + dim)
-            elif stop > dim:
-                stop = dim
-        if start == 0:
-            start = None
-        if stop == dim:
-            stop = None
-        if step == 1:
-            step = None
+        if math.isnan(dim):
+            return idx
+        start, stop, step = idx.indices(dim)
+        if step > 0:
+            if start == 0:
+                start = None
+            if stop >= dim:
+                stop = None
+            if step == 1:
+                step = None
+        elif step < 0:
+            if start >= dim - 1:
+                start = None
+            if stop < 0:
+                stop = None
         return slice(start, stop, step)
     return idx
 


Test patch:
diff --git a/dask/array/tests/test_slicing.py b/dask/array/tests/test_slicing.py
--- a/dask/array/tests/test_slicing.py
+++ b/dask/array/tests/test_slicing.py
@@ -470,6 +470,7 @@ def test_slicing_consistent_names():
     assert same_keys(a[-11:11], a[:])
     assert same_keys(a[-11:-9], a[:1])
     assert same_keys(a[-1], a[9])
+    assert same_keys(a[0::-1], a[0:-11:-1])
 
 
 def test_slicing_consistent_names_after_normalization():
@@ -814,3 +815,8 @@ def test_setitem_with_different_chunks_preserves_shape(params):
     x[mask] = 1
     result = x.compute()
     assert x.shape == result.shape
+
+
+def test_gh3579():
+    assert_eq(np.arange(10)[0::-1], da.arange(10, chunks=3)[0::-1])
+    assert_eq(np.arange(10)[::-1], da.arange(10, chunks=3)[::-1])