internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: dask/dask

Issue:
index-only (no-columns) df.shape issue
``` python
assert dd.from_pandas(pd.DataFrame(index=[1,2,3]), npartitions=1).compute().shape[0] == 3  # that's ok
assert dd.from_pandas(pd.DataFrame(index=[1,2,3]), npartitions=1).shape[0].compute() == 3
```
raises **ValueError: cannot convert float NaN to integer**


Discussion:

Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -3493,6 +3493,8 @@ def shape(self):
         (Delayed('int-07f06075-5ecc-4d77-817e-63c69a9188a8'), 2)
         """
         col_size = len(self.columns)
+        if col_size == 0:
+            return (self.index.shape[0], 0)
         row_size = delayed(int)(self.size / col_size)
         return (row_size, col_size)
 


Test patch:
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -1091,6 +1091,11 @@ def test_shape():
     assert_eq(result[0].compute(), len(full.a))
     assert_eq(dd.compute(result)[0], (len(full.a),))
 
+    sh = dd.from_pandas(pd.DataFrame(index=[1, 2, 3]), npartitions=2).shape
+    assert (sh[0].compute(), sh[1]) == (3, 0)
+    sh = dd.from_pandas(pd.DataFrame({"a": [], "b": []}, index=[]), npartitions=1).shape
+    assert (sh[0].compute(), sh[1]) == (0, 2)
+
 
 def test_nbytes():
     assert_eq(d.a.nbytes, full.a.nbytes)
Repository: dask/dask

Issue:
SeriesGroupBy.nunique not showing up in the docs
https://docs.dask.org/en/latest/dataframe-api.html

I can't seem to click on it either.

![image](https://user-images.githubusercontent.com/17162724/83649353-146ad300-a585-11ea-89fa-3cdc53221bfe.png)



Discussion:
Thanks! It looks like there is no docstring for that method. I am opening a PR now. 

Resolving patch:
diff --git a/dask/dataframe/groupby.py b/dask/dataframe/groupby.py
--- a/dask/dataframe/groupby.py
+++ b/dask/dataframe/groupby.py
@@ -1790,6 +1790,7 @@ def __init__(self, df, by=None, slice=None, **kwargs):
 
         super(SeriesGroupBy, self).__init__(df, by=by, slice=slice, **kwargs)
 
+    @derived_from(pd.core.groupby.SeriesGroupBy)
     def nunique(self, split_every=None, split_out=1):
         name = self._meta.obj.name
         levels = _determine_levels(self.index)


Test patch:
Repository: dask/dask

Issue:
importing fails when calling python -OO
This was discovered by `xarray`'s `upstream-dev` CI ([environment](https://dev.azure.com/xarray/xarray/_build/results?buildId=2996&view=logs&j=2280efed-fda1-53bd-9213-1fa8ec9b4fa8&t=031ddd67-e55f-5fbd-2283-1ff4dfed6587)) a few days ago, but we were a bit slow in reporting so this also happens with the newly released `2.18.0`.

The problem is this:
```
$ python -OO -c 'import dask.array'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File ".../lib/python3.8/site-packages/dask/array/__init__.py", line 26, in <module>
    from .routines import (
  File ".../lib/python3.8/site-packages/dask/array/routines.py", line 18, in <module>
    from .creation import arange, diag, empty, indices
  File ".../lib/python3.8/site-packages/dask/array/creation.py", line 26, in <module>
    from .wrap import empty, ones, zeros, full
  File ".../lib/python3.8/site-packages/dask/array/wrap.py", line 173, in <module>
    full.__name__ = _full.__name__
AttributeError: 'functools.partial' object has no attribute '__name__'
```
without the optimization, the import obviously works.

See also pydata/xarray#4124


Discussion:
Ping @dask/maintenance for visibility. I am taking a look now.

Resolving patch:
diff --git a/dask/array/wrap.py b/dask/array/wrap.py
--- a/dask/array/wrap.py
+++ b/dask/array/wrap.py
@@ -170,6 +170,4 @@ def full_like(a, fill_value, *args, **kwargs):
 
 
 full.__doc__ = _full.__doc__
-full.__name__ = _full.__name__
 full_like.__doc__ = _full_like.__doc__
-full_like.__name__ = _full_like.__name__


Test patch:
Repository: dask/dask

Issue:
Parse Bytes is Incorrect
When using `parse_bytes` with floats less than 1 without a preceding 0 the calculated bytes is incorrect:

```python
In [3]: from dask.utils import parse_bytes

In [4]: parse_bytes('.9GB')
Out[4]: 1900000000

In [5]: parse_bytes('900MB')
Out[5]: 900000000

In [6]: parse_bytes('.9GB') == parse_bytes('900MB')
Out[6]: False

In [7]: parse_bytes('1.1GB') == parse_bytes('1100MB')
Out[7]: True

In [8]: parse_bytes('.5GB')
Out[8]: 1500000000

In [9]: parse_bytes('0.5GB')
Out[9]: 500000000
```

- `.5GB` -> `1500000000`
-  `0.5` -> `500000000`

Probably an addition `1` is getting added here:
https://github.com/dask/dask/blob/dd682ccc81f85d0e6b4782ab131f685ce3b82c89/dask/utils.py#L1214-L1215


Discussion:

Resolving patch:
diff --git a/dask/utils.py b/dask/utils.py
--- a/dask/utils.py
+++ b/dask/utils.py
@@ -1211,7 +1211,7 @@ def parse_bytes(s):
     if isinstance(s, (int, float)):
         return int(s)
     s = s.replace(" ", "")
-    if not s[0].isdigit():
+    if not any(char.isdigit() for char in s):
         s = "1" + s
 
     for i in range(len(s) - 1, -1, -1):


Test patch:
diff --git a/dask/tests/test_utils.py b/dask/tests/test_utils.py
--- a/dask/tests/test_utils.py
+++ b/dask/tests/test_utils.py
@@ -525,6 +525,7 @@ def test_parse_bytes():
     assert parse_bytes("1e6 kB") == 1000000000
     assert parse_bytes("MB") == 1000000
     assert parse_bytes(123) == 123
+    assert parse_bytes(".5GB") == 500000000
 
 
 def test_parse_timedelta():
Repository: dask/dask

Issue:
array.compute_chunk_sizes() returns chunks with numpy integers
I am not sure if this is a bug. If you run this code:
```python

import numpy
import dask.array

random_array = dask.array.from_array(numpy.random.randint(1, 10, size=1000000))
random_array = random_array[random_array > 5]   
random_array.compute_chunk_sizes()
type(random_array.chunks[0][0])
```
The generated chunks data type is numpy.int64. I don't know if this is intentional but it breaks dask.array_to_zarr because zarr tries to encode to json  and  it is not able to deal with numpy data types.

There is an issue in zarr with code to reproduce the error:

https://github.com/zarr-developers/zarr-python/issues/569

- Dask version: 2.18.1
- Python version: 3.8
- Operating System: Linux
- Install method (conda, pip, source): pip



Discussion:
Thanks for the report. Probably not intentional. We can iterate over the `chunk__shape[s]` and get the `.item()` in https://github.com/dask/dask/blob/e7e3b735d4cfee04d44cf7054b122bec1569dd23/dask/array/core.py#L1153 I think.

Any interest in working on this @pziarsolo?
> Thanks for the report. Probably not intentional. We can iterate over the `chunk__shape[s]` and get the `.item()` in
> 
> https://github.com/dask/dask/blob/e7e3b735d4cfee04d44cf7054b122bec1569dd23/dask/array/core.py#L1153
> I think.
> 
> Any interest in working on this @pziarsolo?

I have been looking at the code and I don't feel able to do it.:(



Thanks for taking a look. @gforsyth would you be able to?
I'll take a look, @TomAugspurger -- thanks for the ping.

Resolving patch:
diff --git a/dask/array/core.py b/dask/array/core.py
--- a/dask/array/core.py
+++ b/dask/array/core.py
@@ -1152,7 +1152,11 @@ def compute_chunk_sizes(self):
 
             c.append(tuple(chunk_shapes[s]))
 
-        x._chunks = compute(tuple(c))[0]
+        # `map_blocks` assigns numpy dtypes
+        # cast chunk dimensions back to python int before returning
+        x._chunks = tuple(
+            [tuple([int(chunk) for chunk in chunks]) for chunks in compute(tuple(c))[0]]
+        )
         return x
 
     @property


Test patch:
diff --git a/dask/array/tests/test_array_core.py b/dask/array/tests/test_array_core.py
--- a/dask/array/tests/test_array_core.py
+++ b/dask/array/tests/test_array_core.py
@@ -4214,6 +4214,9 @@ def test_compute_chunk_sizes():
     assert z.chunks == ((10, 10, 5, 0, 0),)
     assert len(z) == 25
 
+    # check that dtype of chunk dimensions is `int`
+    assert isinstance(z.chunks[0][0], int)
+
 
 def test_compute_chunk_sizes_2d_array():
     X = np.linspace(-1, 1, num=9 * 4).reshape(9, 4)
Repository: dask/dask

Issue:
Custom optimization functions are called twice, once without scheduler kwargs.
Custom optimization functions (those given as a scheduler `optimizations=[foo]` kwarg or set in the Dask config) are [being called twice](https://github.com/dask/dask/blob/149abe507a3988073a9acc9d5b8483f44c747f6b/dask/base.py#L223-L224): the first time without scheduler kwargs, the second time with scheduler kwargs.

```python
In [1]: import dask
   ...: import dask.array as da
   ...: import numpy as np

In [2]: def dummy_opt(dsk, keys, **kwargs):
   ...:     print(f"Called with kwargs: {kwargs}")
   ...:     return dsk

In [3]: da.from_array(np.array([1, 2])).compute(optimizations=[dummy_opt], dummy_kwarg="foo")
Called with kwargs: {}
Called with kwargs: {'dummy_kwarg': 'foo'}
Out[3]: array([1, 2])
```

This bug was introduced in [this large performance-oriented PR](https://github.com/dask/dask/pull/6137), and affects the following versions: 2.20.0, 2.19.0, 2.18.1, 2.18.0, 2.17.2, 2.17.1, 2.17.0, and 2.16.0.


Discussion:

Resolving patch:
diff --git a/dask/base.py b/dask/base.py
--- a/dask/base.py
+++ b/dask/base.py
@@ -212,16 +212,18 @@ def collections_to_dsk(collections, optimize_graph=True, **kwargs):
 
         _opt_list = []
         for opt, val in groups.items():
-            _graph_and_keys = _extract_graph_and_keys(val)
-            groups[opt] = _graph_and_keys
-            _opt_list.append(opt(_graph_and_keys[0], _graph_and_keys[1], **kwargs))
+            dsk, keys = _extract_graph_and_keys(val)
+            _opt = opt(dsk, keys, **kwargs)
+            groups[opt] = (_opt, keys)
+            _opt_list.append(_opt)
 
         for opt in optimizations:
             _opt_list = []
             group = {}
             for k, (dsk, keys) in groups.items():
-                group[k] = (opt(dsk, keys), keys)
-                _opt_list.append(opt(dsk, keys, **kwargs))
+                _opt = opt(dsk, keys, **kwargs)
+                group[k] = (_opt, keys)
+                _opt_list.append(_opt)
             groups = group
 
         dsk = merge(*map(ensure_dict, _opt_list,))


Test patch:
Repository: dask/dask

Issue:
dask.base.tokenize raises RecursionError when passed an object that contains cycles
**What happened**: `RecursionError` is raised when passing an object containing cycles to `dask.tokenize`

**What you expected to happen**: A valid token to be returned

**Minimal Complete Verifiable Example**:

```python
import dask.base

cycle = dict(a=None)
cycle["a"] = cycle

dask.base.tokenize(cycle)
```

**Anything else we need to know?**:

**Environment**:

- Dask version: 2.20.0
- Python version: 3.8
- Operating System: Ubuntu 18.04 LTS (running under WSL)
- Install method (conda, pip, source): conda (from conda forge)



Discussion:
Well that is certainly an interesting error! Is there a usecase for this? For context the repr of `cycle` is `{'a': {...}}` maybe dask should tokenize that instead?
We're using bits and pieces of Dask (mostly from `dask.base` and `dask.core`) for manipulating task graphs. At some point we want to tokenize the inputs to a task for the purpose of caching. One of the inputs is a complicated object that happens to contain a cycle, and tokenization fails in this case.

I'm not sure what a solution would look like. IIRC `pickle` solves this problem by keeping track of all the objects it has seen during serialization, perhaps a similar approach could be used here?

Or maybe we should just use a `uuid` if tokenize fails. 
To see if that would work, you could try using tokenize from dask.delayed and setting `pure=False`.

Resolving patch:
diff --git a/dask/base.py b/dask/base.py
--- a/dask/base.py
+++ b/dask/base.py
@@ -697,7 +697,13 @@ def normalize_set(s):
 
 @normalize_token.register((tuple, list))
 def normalize_seq(seq):
-    return type(seq).__name__, list(map(normalize_token, seq))
+    def func(seq):
+        try:
+            return list(map(normalize_token, seq))
+        except RecursionError:
+            return str(uuid.uuid4())
+
+    return type(seq).__name__, func(seq)
 
 
 @normalize_token.register(literal)


Test patch:
diff --git a/dask/tests/test_base.py b/dask/tests/test_base.py
--- a/dask/tests/test_base.py
+++ b/dask/tests/test_base.py
@@ -460,6 +460,13 @@ def test_tokenize_dense_sparse_array(cls_name):
     assert tokenize(a) != tokenize(b)
 
 
+def test_tokenize_object_with_recursion_error_returns_uuid():
+    cycle = dict(a=None)
+    cycle["a"] = cycle
+
+    assert len(tokenize(cycle)) == 32
+
+
 def test_is_dask_collection():
     class DummyCollection(object):
         def __init__(self, dsk=None):
Repository: dask/dask

Issue:
Metadata mismatch is not clear about why columns are different
Reproducible code

```
import pandas as pd
from dask.delayed import delayed
import dask.dataframe as dd

def df_with_str_columns(): 
    return pd.DataFrame({'7': [1, 2], '8': [3, 4]}) 

# Note in meta columns are ints
meta = {7: 'int', 8: 'int'}
df = dd.from_delayed([delayed(df_with_str_columns)(), delayed(df_with_str_columns)()], meta=meta)
df.compute()
```

Output:
```

ValueError: Metadata mismatch found in `from_delayed`.

Partition type: `pandas.core.frame.DataFrame`
+--------+-------+----------+
| Column | Found | Expected |
+--------+-------+----------+
| 7      | int64 | -        |
| 8      | int64 | -        |
| 7      | -     | int64    |
| 8      | -     | int64    |
+--------+-------+----------+
```

The problem is that the output doesn't make it clear that 2 of the columns are strings, while two are ints. The following would be helpful:

```

ValueError: Metadata mismatch found in `from_delayed`.

Partition type: `pandas.core.frame.DataFrame`
+--------+-------+----------+
| Column | Found | Expected |
+--------+-------+----------+
| '7'      | int64 | -        |
| '8'      | int64 | -        |
| 7      | -     | int64    |
| 8      | -     | int64    |
+--------+-------+----------+
```


Discussion:
I can see how this could be moderately infuriating.  Thanks for reporting @birdsarah -- proposed fix incoming.

Resolving patch:
diff --git a/dask/dataframe/utils.py b/dask/dataframe/utils.py
--- a/dask/dataframe/utils.py
+++ b/dask/dataframe/utils.py
@@ -647,7 +647,7 @@ def equal_dtypes(a, b):
     elif is_dataframe_like(meta):
         dtypes = pd.concat([x.dtypes, meta.dtypes], axis=1, sort=True)
         bad_dtypes = [
-            (col, a, b)
+            (repr(col), a, b)
             for col, a, b in dtypes.fillna("-").itertuples()
             if not equal_dtypes(a, b)
         ]


Test patch:
diff --git a/dask/dataframe/tests/test_utils_dataframe.py b/dask/dataframe/tests/test_utils_dataframe.py
--- a/dask/dataframe/tests/test_utils_dataframe.py
+++ b/dask/dataframe/tests/test_utils_dataframe.py
@@ -361,9 +361,9 @@ def test_check_meta():
         "+--------+----------+----------+\n"
         "| Column | Found    | Expected |\n"
         "+--------+----------+----------+\n"
-        "| a      | object   | category |\n"
-        "| c      | -        | float64  |\n"
-        "| e      | category | -        |\n"
+        "| 'a'    | object   | category |\n"
+        "| 'c'    | -        | float64  |\n"
+        "| 'e'    | category | -        |\n"
         "+--------+----------+----------+"
     )
     assert str(err.value) == exp
Repository: dask/dask

Issue:
Cannot pickle dask array objects
**What happened**:

Attempting to pickle a dask array gives:
```
AttributeError: Can't pickle local object 'broadcast_trick.<locals>.inner'
```

Dask arrays are no longer pickleable since v2.23.0.

**What you expected to happen**:

The dask array can be pickled.

**Minimal Complete Verifiable Example**:

```python
import dask.array
import pickle
a = dask.array.ones((10, 10))
pickle.dumps(a)
```

This fails on versions larger than v2.22.

**Anything else we need to know?**:

I am happy to fix this. It seems like a simple fix to move the `inner` function to the module level.

**Environment**:

- Dask version: 2.25.0
- Python version: 3.7.7
- Operating System: Mac OS X
- Install method (conda, pip, source): pip



Discussion:
> It seems like a simple fix to move the inner function to the module level.

That should make things serializable with `pickle`. Though there may be some restrictions on the signature of the function that `broadcast_trick` returns

Alternatively, does using `cloudpickle` work for your use case?
`cloudpickle` does work, so I could probably use it. However if the simple fix works, it would be nice to fix normal pickle.
Ah, sorry I don't think I was very clear in my previous comment. A fix to make things serializable with `pickle` is very welcome. I meant to propose using `cloudpickle` as a workaround. 

Resolving patch:
diff --git a/dask/array/wrap.py b/dask/array/wrap.py
--- a/dask/array/wrap.py
+++ b/dask/array/wrap.py
@@ -135,6 +135,11 @@ def wrap(wrap_func, func, **kwargs):
 w = wrap(wrap_func_shape_as_first_arg)
 
 
+@curry
+def _broadcast_trick_inner(func, shape, *args, **kwargs):
+    return np.broadcast_to(func((), *args, **kwargs), shape)
+
+
 def broadcast_trick(func):
     """
     Provide a decorator to wrap common numpy function with a broadcast trick.
@@ -155,8 +160,7 @@ def broadcast_trick(func):
     so should be safe.
     """
 
-    def inner(shape, *args, **kwargs):
-        return np.broadcast_to(func((), *args, **kwargs), shape)
+    inner = _broadcast_trick_inner(func)
 
     if func.__doc__ is not None:
         inner.__doc__ = func.__doc__


Test patch:
diff --git a/dask/array/tests/test_array_core.py b/dask/array/tests/test_array_core.py
--- a/dask/array/tests/test_array_core.py
+++ b/dask/array/tests/test_array_core.py
@@ -3298,12 +3298,18 @@ def test_from_array_names():
     assert set(names.values()) == set([1, 5])
 
 
-def test_array_picklable():
+@pytest.mark.parametrize(
+    "array",
+    [
+        da.arange(100, chunks=25),
+        da.ones((10, 10), chunks=25),
+    ],
+)
+def test_array_picklable(array):
     from pickle import loads, dumps
 
-    a = da.arange(100, chunks=25)
-    a2 = loads(dumps(a))
-    assert_eq(a, a2)
+    a2 = loads(dumps(array))
+    assert_eq(array, a2)
 
 
 def test_from_array_raises_on_bad_chunks():
Repository: dask/dask

Issue:
reduction "dtype" is required but docstring says optional
**What happened**:

I'm playing around with the dask.array.reduction function while working on a custom algorithm and was confused when I got a ValueError for not specifying the `dtype` when the docstring says it is optional and will default to `x.dtype` if not provided. See:

https://github.com/dask/dask/blob/1d8e1e920b5139bd434749f5877266293f6432a2/dask/array/reductions.py#L145-L146

**What you expected to happen**:

Either do the default that the docstring says or make dtype a required argument.

**Minimal Complete Verifiable Example**:

I can make one if you really need it, but this is more of a behavioral question about what is actually desired. Is defaulting to the input array's dtype going to cause issues in the future (has it happened in other dask functions)? If so, I'll update the docstring and make a PR. If not, I'll update the code to default to `x.dtype`.


Discussion:
Thanks @djhoese for reporting this. I think if dtype isn't provided it should be inferred from doing the operation on `meta` like `map_blocks` does. As far as I can tell, this mandatory `dtype` was implemented in #1755, so I'm pinging @jcrist in case he has input on what the behavior should be.

Resolving patch:
diff --git a/dask/array/reductions.py b/dask/array/reductions.py
--- a/dask/array/reductions.py
+++ b/dask/array/reductions.py
@@ -77,8 +77,9 @@ def reduction(
     keepdims: boolean, optional
         Whether the reduction function should preserve the reduced axes,
         leaving them at size ``output_size``, or remove them.
-    dtype: np.dtype, optional
-        Force output dtype. Defaults to x.dtype if omitted.
+    dtype: np.dtype
+        data type of output. This argument was previously optional, but
+        leaving as ``None`` will now raise an exception.
     split_every: int >= 2 or dict(axis: int), optional
         Determines the depth of the recursive aggregation. If set to or more
         than the number of input chunks, the aggregation will be performed in


Test patch:
Repository: dask/dask

Issue:
Fastavro iter_avro
Im trying to use the `bag.read_avro()` functionality, but I get the following error:

```
AttributeError: module 'fastavro' has no attribute 'iter_avro'
```

Here is the code to reproduce the case:

```
import dask.bag as db
path_input = './data/*.avro'
df = db.read_avro(path_input).compute()
```

From the fastavro (1.0.0.post1) doc, I get it should be calling `fastavro.reader` instead of `fastavro.iter_avro`:
https://github.com/fastavro/fastavro/blob/3474b6c8a2a38c7d39ca955a5e3d0261d6f3659f/fastavro/read.py

**Environment**:

- Dask version: 2.11.0
- Fastavro version: 1.0.0.post1
- Python version: 3.7
- Operating System: Linux
- Install method (conda, pip, source): pip



Discussion:
Thanks for the report. IIUC, `fastavro` has changed their API? Do you want to update https://github.com/dask/dask/blob/7f5f7fe49298c2353977b99a081be47dc28bc358/dask/bag/avro.py to be compatible with it (and the old version too, ideally).
Ref: https://github.com/dask/dask/pull/6740
@lqrz you gave a thumbsup to https://github.com/dask/dask/issues/6744#issuecomment-712036919. Does that mean you're looking into updating the code in dask.bag.avro?
@TomAugspurger Hi, Tom! nono, I just intended to support your good interpretation of my original report 😄 ; but im not looking into it!

Resolving patch:
diff --git a/dask/bag/avro.py b/dask/bag/avro.py
--- a/dask/bag/avro.py
+++ b/dask/bag/avro.py
@@ -141,21 +141,31 @@ def read_chunk(fobj, off, l, head):
     import fastavro
     from dask.bytes.core import read_block
 
+    if hasattr(fastavro, "iter_avro"):
+        reader = fastavro.iter_avro
+    else:
+        reader = fastavro.reader
+
     with fobj as f:
         chunk = read_block(f, off, l, head["sync"])
     head_bytes = head["head_bytes"]
     if not chunk.startswith(MAGIC):
         chunk = head_bytes + chunk
     i = io.BytesIO(chunk)
-    return list(fastavro.iter_avro(i))
+    return list(reader(i))
 
 
 def read_file(fo):
     """Get rows from file-like"""
     import fastavro
 
+    if hasattr(fastavro, "iter_avro"):
+        reader = fastavro.iter_avro
+    else:
+        reader = fastavro.reader
+
     with fo as f:
-        return list(fastavro.iter_avro(f))
+        return list(reader(f))
 
 
 def to_avro(


Test patch: