internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: dask/dask

Issue:
apply_along_axis infers output shape improperly
My environment:
Python: 3.6.6 |Anaconda custom (64-bit)
Dask: 0.18.1

This fails:
```python
import dask.array as da
def testfun(v):
    return v[0] + v[1]
dar = da.random.randint(0,1, size=(2,2), chunks=('auto'))
da.apply_along_axis(testfun, 0, dar)
```

```pytb
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-2-43967c9b007c> in <module>
      3     return v[0] + v[1]
      4 dar = da.random.randint(0,1, size=(2,2), chunks=('auto'))
----> 5 da.apply_along_axis(testfun, 0, dar)

~/sandbox/dask/dask/array/routines.py in apply_along_axis(func1d, axis, arr, *args, **kwargs)
    321     # Test out some data with the function.
    322     test_data = np.ones((1,), dtype=arr.dtype)
--> 323     test_result = np.array(func1d(test_data, *args, **kwargs))
    324
    325     # Rechunk so that func1d is applied over the full axis.

<ipython-input-2-43967c9b007c> in testfun(v)
      1 import dask.array as da
      2 def testfun(v):
----> 3     return v[0] + v[1]
      4 dar = da.random.randint(0,1, size=(2,2), chunks=('auto'))
      5 da.apply_along_axis(testfun, 0, dar)

IndexError: index 1 is out of bounds for axis 0 with size 1
```

The reason why is because of [this line](https://github.com/dask/dask/blob/36e235d7074f01830dd79a24fd656f4cda9e2c81/dask/array/routines.py#L324) in `apply_along_axis`, where the function arg in `apply_along_axis` is applied to non-representative test data (`np.ones((1,))`) for estimating output dimensionality and dtype.

At the least, the properties of the output being inferred here (shape and dtype) should be specifiable as kwargs to `apply_along_axis` so the user can dodge having them inferred. 

If these properties are going to be inferred, the sample data fed into the test function should at least match the size (and probably the dtype) of the vectors the user wants to feed `apply_along_axis` .

If I have the time I can study how this is done for comparable functions (`map_blocks` comes to mind) and submit a PR, but I may not have the time soon. 


Discussion:

Resolving patch:
diff --git a/dask/array/core.py b/dask/array/core.py
--- a/dask/array/core.py
+++ b/dask/array/core.py
@@ -2018,7 +2018,7 @@ def normalize_chunks(chunks, shape=None, limit=None, dtype=None,
     """ Normalize chunks to tuple of tuples
 
     This takes in a variety of input types and information and produces a full
-    tuple-of-tuples result for chunks, sutiable to be passed to Array or
+    tuple-of-tuples result for chunks, suitable to be passed to Array or
     rechunk or any other operation that creates a Dask array.
 
     Parameters
@@ -3734,7 +3734,7 @@ def stack(seq, axis=0):
     """
     Stack arrays along a new axis
 
-    Given a sequence of dask Arrays form a new dask Array by stacking them
+    Given a sequence of dask arrays, form a new dask array by stacking them
     along a new dimension (axis=0 by default)
 
     Examples


Test patch:
Repository: dask/dask

Issue:
Dask array setitem with differently chunked data provides wrong shape
```python
>>> import dask.array as da
>>> x = da.zeros(5, chunks=2)
>>> m = da.zeros(5, chunks=3)
>>> x[m>0] = 1
>>> print x.shape, x.compute().shape
(5,) (4,)
```

This should probably unifiy chunks before slicing


Discussion:
Marking this as "good first issue".  I suspect that it might be a bit challenging, but it does take new developers through thinking about chunks.
Looks educational, going to try to do this.

Resolving patch:
diff --git a/dask/array/core.py b/dask/array/core.py
--- a/dask/array/core.py
+++ b/dask/array/core.py
@@ -1337,6 +1337,7 @@ def __setitem__(self, key, value):
             self.dtype = y.dtype
             self.dask = y.dask
             self.name = y.name
+            self._chunks = y.chunks
             return self
         else:
             raise NotImplementedError("Item assignment with %s not supported"


Test patch:
diff --git a/dask/array/tests/test_slicing.py b/dask/array/tests/test_slicing.py
--- a/dask/array/tests/test_slicing.py
+++ b/dask/array/tests/test_slicing.py
@@ -798,3 +798,19 @@ def test_pathological_unsorted_slicing():
 
     assert '10' in str(info.list[0])
     assert 'out-of-order' in str(info.list[0])
+
+
+@pytest.mark.parametrize('params', [(2, 2, 1), (5, 3, 2)])
+def test_setitem_with_different_chunks_preserves_shape(params):
+    """ Reproducer for https://github.com/dask/dask/issues/3730.
+
+    Mutating based on an array with different chunks can cause new chunks to be
+    used.  We need to ensure those new chunk sizes are applied to the mutated
+    array, otherwise the array won't generate the correct keys.
+    """
+    array_size, chunk_size1, chunk_size2 = params
+    x = da.zeros(array_size, chunks=chunk_size1)
+    mask = da.zeros(array_size, chunks=chunk_size2)
+    x[mask] = 1
+    result = x.compute()
+    assert x.shape == result.shape
Repository: dask/dask

Issue:
Applying da.arg* with axis argument uses non-tuple nd indexing
Applying a `da.array` function such as `da.argmax` along an axis yields the following warning:

```
C:\ProgramData\Anaconda3\lib\site-packages\dask\array\reductions.py:564:` FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. 
In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  vals = vals[inds]
```
This is not triggered for a function such as `da.max` and not triggered if either no axis argument is given or the array is 1D.
Minimal working example:
```
import dask array as da

Y =  da.random.random((10,10), chunks='auto')
da.argmax(Y, axis=0).compute()
```

Shows up on Python3.6 since I updated from dask 0.18.1 to 0.18.2.

I would like to spend some time on fixing this to get into this as my first issue, but frankly, I have no idea where to start.



Discussion:
Thanks for the report.

> Shows up on Python3.6 since I updated from dask 0.18.1 to 0.18.2.

I suspect that that upgrade also triggered an update to NumPy 1.15, which IIRC introduced this warning.

When debugging these warnings, I find elevating them to errors helpful

```python
import warnings
warnings.simplefilter("error", FutureWarning)
```

This points to https://github.com/dask/dask/blob/master/dask/array/reductions.py#L562-L565 as the source of the problem. We may be able to make `inds` a tuple, but I haven't looked closely. Let us know if you have any issues debugging further!
Ah yes, the warning trace already displayed that now I look at it. I only today read the relevant NEP, I will have a closer look this weekend.

Resolving patch:
diff --git a/dask/array/reductions.py b/dask/array/reductions.py
--- a/dask/array/reductions.py
+++ b/dask/array/reductions.py
@@ -532,6 +532,7 @@ def _arg_combine(data, axis, argfunc, keepdims=False):
         local_args = argfunc(vals, axis=axis)
         inds = np.ogrid[tuple(map(slice, local_args.shape))]
         inds.insert(axis, local_args)
+        inds = tuple(inds)
         vals = vals[inds]
         arg = arg[inds]
         if keepdims:


Test patch:
diff --git a/dask/array/tests/test_array_core.py b/dask/array/tests/test_array_core.py
--- a/dask/array/tests/test_array_core.py
+++ b/dask/array/tests/test_array_core.py
@@ -3620,3 +3620,11 @@ def test_dask_array_holds_scipy_sparse_containers():
     zz = z.compute(scheduler='single-threaded')
     assert isinstance(yy, scipy.sparse.spmatrix)
     assert (zz == xx.T).all()
+
+
+def test_3851():
+    with warnings.catch_warnings() as record:
+        Y = da.random.random((10, 10), chunks='auto')
+        da.argmax(Y, axis=0).compute()
+
+    assert not record
Repository: dask/dask

Issue:
categorical.as_known returns incorrect type for already known categorical
In https://github.com/dask/dask/blob/003962f61dc2c19e363ad723e9bb0ca406afd621/dask/dataframe/categorical.py#L186-L187 we return `self`, which is a `CategoricalAccessor`.

I think that should return `self._series`, the dask series.


Discussion:
hey @TomAugspurger will take up this.

Resolving patch:
diff --git a/dask/dataframe/categorical.py b/dask/dataframe/categorical.py
--- a/dask/dataframe/categorical.py
+++ b/dask/dataframe/categorical.py
@@ -184,7 +184,7 @@ def as_known(self, **kwargs):
             Keywords to pass on to the call to `compute`.
         """
         if self.known:
-            return self
+            return self._series
         categories = self._property_map('categories').unique().compute(**kwargs)
         return self.set_categories(categories.values)
 


Test patch:
diff --git a/dask/dataframe/tests/test_categorical.py b/dask/dataframe/tests/test_categorical.py
--- a/dask/dataframe/tests/test_categorical.py
+++ b/dask/dataframe/tests/test_categorical.py
@@ -271,6 +271,14 @@ def assert_array_index_eq(left, right):
     assert_eq(left, pd.Index(right) if isinstance(right, np.ndarray) else right)
 
 
+def test_return_type_known_categories():
+    df = pd.DataFrame({"A": ['a', 'b', 'c']})
+    df['A'] = df['A'].astype('category')
+    dask_df = dd.from_pandas(df, 2)
+    ret_type = dask_df.A.cat.as_known()
+    assert isinstance(ret_type, dd.core.Series)
+
+
 class TestCategoricalAccessor:
 
     @pytest.mark.parametrize('series', cat_series)
Repository: dask/dask

Issue:
Provide _ipython_key_completions_
PR coming, for example

```python
import pandas as pd
import dask.dataframe as dd

In [2]: df = pd.DataFrame({'a_long_column': [1, 2, 3]})

In [3]: df['a_'<TAB>
In [3]: df['a_long_column

In [5]: ddf = dd.from_pandas(df, npartitions=1)

In [4]: ddf['a_'<TAB>  # no completion
```



Discussion:

Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -2527,6 +2527,9 @@ def __dir__(self):
                   pd.compat.isidentifier(c)))
         return list(o)
 
+    def _ipython_key_completions_(self):
+        return self.columns.tolist()
+
     @property
     def ndim(self):
         """ Return dimensionality """


Test patch:
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -2718,6 +2718,16 @@ def test_getitem_column_types(col_type):
     assert_eq(df[cols], ddf[cols])
 
 
+def test_ipython_completion():
+    df = pd.DataFrame({'a': [1], 'b': [2]})
+    ddf = dd.from_pandas(df, npartitions=1)
+
+    completions = ddf._ipython_key_completions_()
+    assert 'a' in completions
+    assert 'b' in completions
+    assert 'c' not in completions
+
+
 def test_diff():
     df = pd.DataFrame(np.random.randn(100, 5), columns=list('abcde'))
     ddf = dd.from_pandas(df, 5)
Repository: dask/dask

Issue:
Bag groupby error using tasks based shuffle
Looks like the issue is when the bag in question has only 1 partition.

```
>>> dask.bag.from_sequence('abc', npartitions = 1).groupby(lambda x: x, shuffle = 'tasks').compute()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/dask/bag/core.py", line 1240, in groupby
    return groupby_tasks(self, grouper, max_branch=max_branch)
  File "/usr/local/lib/python3.6/dist-packages/dask/bag/core.py", line 2012, in groupby_tasks
    for i, j in enumerate(join))
UnboundLocalError: local variable 'join' referenced before assignment
```

Dask version 0.18.1


Discussion:
Thanks @mohtar for the opening up this issue! I was able to reproduce this error on the current `master` branch. Do you have an interest in opening up a PR? 

Resolving patch:
diff --git a/dask/bag/core.py b/dask/bag/core.py
--- a/dask/bag/core.py
+++ b/dask/bag/core.py
@@ -1970,7 +1970,7 @@ def groupby_tasks(b, grouper, hash=hash, max_branch=32):
     max_branch = max_branch or 32
     n = b.npartitions
 
-    stages = int(math.ceil(math.log(n) / math.log(max_branch)))
+    stages = int(math.ceil(math.log(n) / math.log(max_branch))) or 1
     if stages > 1:
         k = int(math.ceil(n ** (1 / stages)))
     else:


Test patch:
diff --git a/dask/bag/tests/test_bag.py b/dask/bag/tests/test_bag.py
--- a/dask/bag/tests/test_bag.py
+++ b/dask/bag/tests/test_bag.py
@@ -1138,7 +1138,8 @@ def test_groupby_tasks_names():
 
 
 @pytest.mark.parametrize('size,npartitions,groups', [(1000, 20, 100),
-                                                     (12345, 234, 1042)])
+                                                     (12345, 234, 1042),
+                                                     (100, 1, 50)])
 def test_groupby_tasks_2(size, npartitions, groups):
     func = lambda x: x % groups
     b = db.range(size, npartitions=npartitions).groupby(func, shuffle='tasks')
Repository: dask/dask

Issue:
Indexing a dask array with a decreasing slice results in negative dimension size
On dask 0.16:
```
In [39]: import dask.array as da

In [40]: x = da.ones(10, chunks=-1)

# notice the negative shape!
In [41]: x[6:3]
Out[41]: dask.array<getitem, shape=(-3,), dtype=float64, chunksize=(-3,)>

# at least it gets computed correctly.
In [42]: x[6:3].compute()
Out[42: array([], dtype=float64)
```

This manifested itself in an issue for xarray: https://github.com/pydata/xarray/issues/1778


Discussion:
That's clearly wrong. Do you know if this has always been the case or if it was this some kind of regression?
Not sure. It could have been here for a while -- decreasing slices are pretty rare, they aren't actually useful for anything.
I ran into the same error as pydata/xarray#1778 but using slice(None, None, -1) to revert an axis. There is no error when i run it in dask version 0.15.0,

Resolving patch:
diff --git a/dask/array/slicing.py b/dask/array/slicing.py
--- a/dask/array/slicing.py
+++ b/dask/array/slicing.py
@@ -728,6 +728,8 @@ def normalize_slice(idx, dim):
                 stop = None
             if step == 1:
                 step = None
+            if stop is not None and start is not None and stop < start:
+                stop = start
         elif step < 0:
             if start >= dim - 1:
                 start = None


Test patch:
diff --git a/dask/array/tests/test_array_core.py b/dask/array/tests/test_array_core.py
--- a/dask/array/tests/test_array_core.py
+++ b/dask/array/tests/test_array_core.py
@@ -3593,3 +3593,10 @@ def test_atop_large_inputs_delayed():
     d = atop(lambda x, y: x + y, 'i', a, 'i', y=b, dtype=a.dtype)
     assert any(b is v for v in d.dask.values())
     assert repr(dict(c.dask)).count(repr(b)[:10]) == 1  # only one occurrence
+
+
+def test_slice_reversed():
+    x = da.ones(10, chunks=-1)
+    y = x[6:3]
+
+    assert_eq(y, np.ones(0))
Repository: dask/dask

Issue:
Azure datalake support in dask
[azure.datalake.store](https://github.com/Azure/azure-data-lake-store-python) provides pythonic access to bytes stored on Azure.

The following file provides a shim to enable dask to access azure in parallel, given urls like `'adl://bucket/path/to/key'`: https://github.com/oxygenanywhere/dask/blob/master/dask/bytes/adl.py , putting it on par with hdfs, s3, gcs.

This module is anecdotally believed to be working. However, we currently have no way to test this, since we have no Azure credentials. Furthermore, it's not clear where the module should live. If within dask itself, then some testing machinery (such as VCR) would need to be pulled in. Perhaps the module would find a more obvious place in the ADL repo.

Currently, the filesystem instance will pass credential strings between processes, which is potentially insecure - although perhaps this is less severe for applications presumed to be running within Azure. It may be better to think of the best way to securely provide a source of credentials to each worker.

cc @teoliphant 

xref #2776 


Discussion:
@martindurant can I ask you to provide an entry in the docs pointing to the adl support, perhaps with a suitable "experimental" note?  That way we can close this out, and hopefully get more eyes on the project.
Yes, will do. For reference, https://github.com/dask/dask-adlfs is what you need.
(that shim could live in fsspec in the future...)

Resolving patch:
diff --git a/dask/bytes/core.py b/dask/bytes/core.py
--- a/dask/bytes/core.py
+++ b/dask/bytes/core.py
@@ -476,6 +476,19 @@ def get_fs(protocol, storage_options=None):
                         "    pip install gcsfs")
         cls = _filesystems[protocol]
 
+    elif protocol in ['adl', 'adlfs']:
+
+        import_required('dask_adlfs',
+                        "Need to install `dask_adlfs` for Azure Datalake "
+                        "Storage support.\n"
+                        "First install azure-storage via pip or conda:\n"
+                        "    conda install -c conda-forge azure-storage\n"
+                        "    or\n"
+                        "    pip install azure-storage\n"
+                        "and then install `dask_adlfs` via pip:\n"
+                        "    pip install dask-adlfs")
+
+        cls = _filesystems[protocol]
     elif protocol == 'hdfs':
         cls = get_hdfs_driver(config.get("hdfs_driver", "auto"))
 


Test patch:
Repository: dask/dask

Issue:
Passing a tuple/namedtuple as a kwarg to atop/top fails on dask 0.20.0
but used to work on dask 0.19.4

```python
from collections import namedtuple

import dask
import dask.array as da


A = da.random.random((100, 100), chunks=(10, 10))

foo = namedtuple("foo", ["bar", "qux"])


def wrappymcwrapface(data, fooval):
    return data


result = da.atop(wrappymcwrapface, ("d1", "d2"),
                 A, ("d1", "d2"),
                 fooval=foo("pants", "marmalade"),
                 dtype=A.dtype)
```

The above produces the following backtrace.

```backtrace
$ python test_dask_namedtuple_kwargs.py 
Traceback (most recent call last):
  File "test_dask_namedtuple_kwargs.py", line 19, in <module>
    dtype=A.dtype)
  File "/home/sperkins/venv/dask-0.20.0/local/lib/python2.7/site-packages/dask/array/top.py", line 482, in atop
    dsk = _top(func, out, out_ind, *argindsstr, numblocks=numblocks, **kwargs2)
  File "/home/sperkins/venv/dask-0.20.0/local/lib/python2.7/site-packages/dask/array/top.py", line 96, in _top
    kwargs = subs(kwargs, sub)
  File "/home/sperkins/venv/dask-0.20.0/local/lib/python2.7/site-packages/dask/array/top.py", line 22, in subs
    return type(task)([subs(x, substitution) for x in task])
  File "/home/sperkins/venv/dask-0.20.0/local/lib/python2.7/site-packages/dask/array/top.py", line 22, in subs
    return type(task)([subs(x, substitution) for x in task])
  File "/home/sperkins/venv/dask-0.20.0/local/lib/python2.7/site-packages/dask/array/top.py", line 22, in subs
    return type(task)([subs(x, substitution) for x in task])
  File "/home/sperkins/venv/dask-0.20.0/local/lib/python2.7/site-packages/dask/array/top.py", line 22, in subs
    return type(task)([subs(x, substitution) for x in task])
TypeError: __new__() takes exactly 3 arguments (2 given)
```


Discussion:

Resolving patch:
diff --git a/dask/array/top.py b/dask/array/top.py
--- a/dask/array/top.py
+++ b/dask/array/top.py
@@ -18,7 +18,7 @@ def subs(task, substitution):
     """
     if isinstance(task, dict):
         return {k: subs(v, substitution) for k, v in task.items()}
-    if isinstance(task, (tuple, list, set)):
+    if type(task) in (tuple, list, set):
         return type(task)([subs(x, substitution) for x in task])
     try:
         return substitution[task]


Test patch:
diff --git a/dask/array/tests/test_atop.py b/dask/array/tests/test_atop.py
--- a/dask/array/tests/test_atop.py
+++ b/dask/array/tests/test_atop.py
@@ -1,3 +1,4 @@
+import collections
 from operator import add
 
 import pytest
@@ -361,3 +362,21 @@ def test_svd():
     u, s, v = da.linalg.svd(y)
     z = y + u
     assert_eq(z, z)
+
+
+@pytest.mark.parametrize('tup', [
+    (1, 2),
+    collections.namedtuple('foo', ['a', 'b'])(1, 2),
+])
+def test_namedtuple(tup):
+    A = da.random.random((20, 20), chunks=(10, 10))
+
+    def f(data, x):
+        return data
+
+    B = da.atop(f, ("d1", "d2"),
+                A, ("d1", "d2"),
+                x=tup,
+                dtype=A.dtype)
+
+    assert_eq(A, B)
Repository: dask/dask

Issue:
tensordot cannot handle >26 dimensions
Currently ``tensordot`` is limited to tensors with 26 or fewer dimensions. Reproducing example:

```python
>>> import dask.array as da

>>> x = da.random.random([2]*26, chunks=-1)
>>> da.tensordot(x, x, 26)  # works fine
dask.array<sum-aggregate, shape=(), dtype=float64, chunksize=()>

>>> x = da.random.random([2]*27, chunks=-1)
>>> da.tensordot(x, x, 27)  # errors
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/johnnie/conda/lib/python3.6/site-packages/dask/array/routines.py", line 235, in tensordot
    right_index[r] = left_index[l]
IndexError: list index out of range
```

The culprit is these lines in ``dask/array/routines.py::tensordot``
```python
    ...
    left_index = list(alphabet[:lhs.ndim])
    right_index = list(ALPHABET[:rhs.ndim])
    ...
```
Which silently slices too few characters, which are then supplied to ``atop``.

I know this sounds like quite an niche use-case! But it would actually be super useful to support this (e.g. in quantum physics where such arrays appear often) as this is the exact kind of size where memory limitations make dask's array chunking attractive/necessary.



Discussion:
Seems like a reasonable request to me.  Any interest in submitting a PR to help resolve this @jcmgray ?  
Sure thing, I'll have a deeper look at a fix momentarily. 

Resolving patch:
diff --git a/dask/array/routines.py b/dask/array/routines.py
--- a/dask/array/routines.py
+++ b/dask/array/routines.py
@@ -226,8 +226,8 @@ def tensordot(lhs, rhs, axes=2):
 
     dt = np.promote_types(lhs.dtype, rhs.dtype)
 
-    left_index = list(alphabet[:lhs.ndim])
-    right_index = list(ALPHABET[:rhs.ndim])
+    left_index = list(range(lhs.ndim))
+    right_index = list(range(lhs.ndim, lhs.ndim + rhs.ndim))
     out_index = left_index + right_index
 
     for l, r in zip(left_axes, right_axes):


Test patch:
diff --git a/dask/array/tests/test_routines.py b/dask/array/tests/test_routines.py
--- a/dask/array/tests/test_routines.py
+++ b/dask/array/tests/test_routines.py
@@ -287,6 +287,13 @@ def test_tensordot_2(axes):
               np.tensordot(x, x, axes=axes))
 
 
+def test_tensordot_more_than_26_dims():
+    ndim = 27
+    x = np.broadcast_to(1, [2] * ndim)
+    dx = da.from_array(x, chunks=-1)
+    assert_eq(da.tensordot(dx, dx, ndim), np.array(2**ndim))
+
+
 def test_dot_method():
     x = np.arange(400).reshape((20, 20))
     a = da.from_array(x, chunks=(5, 5))