internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: dask/dask

Issue:
Feature request: improve error reporting when misusing da.histogram
When porting code like this:

```python
hist, edges = np.histogram(data, bins=40)
```

to dask, it's easy to do:

```python
hist, edges = da.histogram(data, bins=40)
```

and forget to specify a range.

The error that you get in this case is:

```python-traceback
~/miniconda3/envs/ovscrptd/lib/python3.6/site-packages/dask/array/routines.py in histogram(a, bins, range, normed, weights, density)
    612     if not np.iterable(bins):
    613         bin_token = bins
--> 614         mn, mx = range
    615         if mn == mx:
    616             mn -= 0.5

TypeError: 'NoneType' object is not iterable
```

An extra check here and a nudge to add a range or specify bins would be helpful, specially for noobs like me.


Discussion:

Resolving patch:
diff --git a/dask/array/routines.py b/dask/array/routines.py
--- a/dask/array/routines.py
+++ b/dask/array/routines.py
@@ -598,9 +598,10 @@ def histogram(a, bins=None, range=None, normed=False, weights=None, density=None
     >>> h.compute()
     array([5000, 5000])
     """
-    if bins is None or (range is None and bins is None):
-        raise ValueError('dask.array.histogram requires either bins '
-                         'or bins and range to be defined.')
+    if not np.iterable(bins) and (range is None or bins is None):
+        raise ValueError('dask.array.histogram requires either specifying '
+                         'bins as an iterable or specifying both a range and '
+                         'the number of bins')
 
     if weights is not None and weights.chunks != a.chunks:
         raise ValueError('Input array and weights must have the same '


Test patch:
diff --git a/dask/array/tests/test_routines.py b/dask/array/tests/test_routines.py
--- a/dask/array/tests/test_routines.py
+++ b/dask/array/tests/test_routines.py
@@ -601,6 +601,20 @@ def test_histogram_extra_args_and_shapes():
                   da.histogram(v, bins=bins, weights=w, density=True)[0])
 
 
+@pytest.mark.parametrize("bins, hist_range", [
+    (None, None),
+    (10, None),
+    (None, (1, 10)),
+])
+def test_histogram_bin_range_raises(bins, hist_range):
+    data = da.random.random(10, chunks=2)
+    with pytest.raises(ValueError) as info:
+        da.histogram(data, bins=bins, range=hist_range)
+    err_msg = str(info.value)
+    assert 'bins' in err_msg
+    assert 'range' in err_msg
+
+
 def test_cov():
     x = np.arange(56).reshape((7, 8))
     d = da.from_array(x, chunks=(4, 4))
Repository: dask/dask

Issue:
"got an unexpected keyword argument 'username'" when reading from hdfs with authorization
```
  File "x.py", line 4, in <module>
    dd.read_csv('hdfs://user@example.org/file.csv')
  File "/Users/r/proj/env/lib/python3.7/site-packages/dask/dataframe/io/csv.py", line 488, in read
    **kwargs)
  File "/Users/r/proj/env/lib/python3.7/site-packages/dask/dataframe/io/csv.py", line 343, in read_pandas
    **(storage_options or {}))
  File "/Users/r/proj/env/lib/python3.7/site-packages/dask/bytes/core.py", line 80, in read_bytes
    storage_options=kwargs)
  File "/Users/r/proj/env/lib/python3.7/site-packages/dask/bytes/core.py", line 354, in get_fs_token_paths
    fs, fs_token = get_fs(protocol, options)
  File "/Users/r/proj/env/lib/python3.7/site-packages/dask/bytes/core.py", line 513, in get_fs
    fs = cls(**storage_options)
  File "/Users/r/proj/env/lib/python3.7/site-packages/dask/bytes/pyarrow.py", line 28, in __init__
    self.fs = pa.hdfs.HadoopFileSystem(**kwargs)
TypeError: __init__() got an unexpected keyword argument 'username'
```

where x.py is:

```
import dask.dataframe as dd
dd.read_csv('hdfs://user@example.org/file.csv')
```

Dask version: '1.1.0'.

Pyarrow version: '0.12.0'.

Seems like bytes/utils.py extract username as "username" (https://github.com/dask/dask/blob/master/dask/bytes/utils.py#L81), while pyarrow expects "user" (https://github.com/apache/arrow/blob/master/python/pyarrow/hdfs.py#L205). It also seems like in both places those keys were used for a while now. 



Discussion:
This may be an artefact from switching from hdfs3 to arrow's hdfs.
To work around, you could provide the URL as `'hdfs://example.org/file.csv'` and `storage_options={'user': 'user'}`.
(a PR to rename username->user for use with HDFS, and a test which verifies that it works, would be very welcome)

Resolving patch:
diff --git a/dask/bytes/pyarrow.py b/dask/bytes/pyarrow.py
--- a/dask/bytes/pyarrow.py
+++ b/dask/bytes/pyarrow.py
@@ -21,11 +21,18 @@ def isfile(self, path):
 _MIN_PYARROW_VERSION_SUPPORTED = '0.8.1.dev81'
 
 
+def update_hdfs_options(options):
+    username = options.pop('username', None)
+    if username is not None:
+        options['user'] = username
+    return options
+
+
 class PyArrowHadoopFileSystem(object):
     sep = "/"
 
     def __init__(self, **kwargs):
-        self.fs = pa.hdfs.HadoopFileSystem(**kwargs)
+        self.fs = pa.hdfs.HadoopFileSystem(update_hdfs_options(**kwargs))
 
     @classmethod
     def from_pyarrow(cls, fs):


Test patch:
Repository: dask/dask

Issue:
yaml.load() deprecation warning after updating to pyyaml v5.1 (released 13/03/19)
Very small issue. Wanted to flag early. Just updated to the lastest version of pyyaml (a dependency of dask) v5.1. After updating I get the following deprecation warning. 

```
/dask/config.py:168: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  data = yaml.load(f.read()) or {}
```


Discussion:
Yep, there's a distributed issue as well. A PR updating to the safe method
loader would be welcome.

On Thu, Mar 14, 2019 at 12:19 PM kindjacket <notifications@github.com>
wrote:

> Very small issue. Wanted to flag early. Just updated to the lastest
> version of pyyaml (a dependency of dask) v5.1. After updating I get the
> following deprecation warning.
>
>   data = yaml.load(f.read()) or {}```
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/4597>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIsfX5-EiLYHSwv1DJYZCpo2OoBq0ks5vWoSogaJpZM4b0yvI>
> .
>


Resolving patch:
diff --git a/dask/config.py b/dask/config.py
--- a/dask/config.py
+++ b/dask/config.py
@@ -165,7 +165,7 @@ def collect_yaml(paths=paths):
     for path in file_paths:
         try:
             with open(path) as f:
-                data = yaml.load(f.read()) or {}
+                data = yaml.safe_load(f.read()) or {}
                 data = normalize_nested_keys(data)
                 configs.append(data)
         except (OSError, IOError):


Test patch:
Repository: dask/dask

Issue:
deprecation warnings in read_csv
The use of `pd.Categorical.from_codes` cases the following warnings:
```
intake/source/tests/test_csv.py::test_read_pattern
  /Users/mdurant/code/dask/dask/dataframe/io/csv.py:252: FutureWarning: float codes will be disallowed in the future and raise a ValueError
    colname: pd.Categorical.from_codes(np.zeros(len(head)), paths)

intake/source/tests/test_csv.py::test_read_pattern_with_path_as_pattern_str
  /Users/mdurant/code/dask/dask/dataframe/io/csv.py:252: FutureWarning: float codes will be disallowed in the future and raise a ValueError
    colname: pd.Categorical.from_codes(np.zeros(len(head)), paths)
```
Presumably fixed by specifying an int type to the numpy call.
( cc @jsignell )


Discussion:
Don't know how I missed this. I'll make a PR.

Resolving patch:
diff --git a/dask/dataframe/io/csv.py b/dask/dataframe/io/csv.py
--- a/dask/dataframe/io/csv.py
+++ b/dask/dataframe/io/csv.py
@@ -249,7 +249,8 @@ def text_blocks_to_pandas(reader, block_lists, header, head, kwargs,
     if collection:
         if path:
             head = head.assign(**{
-                colname: pd.Categorical.from_codes(np.zeros(len(head)), paths)
+                colname: pd.Categorical.from_codes(
+                    np.zeros(len(head), dtype=int), paths)
             })
         if len(unknown_categoricals):
             head = clear_known_categories(head, cols=unknown_categoricals)


Test patch:
Repository: dask/dask

Issue:
Dask Dataframe best practices
I'd like for us to make a doc page about best practices for Dask DataFrame. What are some topics?

As a reminder, here is the best practices doc for dask delayed: http://docs.dask.org/en/latest/delayed-best-practices.html

Here is a current page that we might update: https://docs.dask.org/en/latest/dataframe-performance.html and bring in line with common styling

What else should we mention that isn't in that page?  cc @jrbourbeau @TomAugspurger @jcrist  


Discussion:
There's some content in a few other pages that may full under best practices as well:

- https://docs.dask.org/en/latest/dataframe-design.html (partitions, passing metadata)
- https://docs.dask.org/en/latest/dataframe-categoricals.html (how to use known/unknown categoricals)
- https://docs.dask.org/en/latest/dataframe.html#common-uses-and-anti-uses

Resolving patch:
diff --git a/docs/source/conf.py b/docs/source/conf.py
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -311,6 +311,7 @@
     ('array-overview.html', 'array.html'),
     ('array-ghost.html', 'array-overlap.html'),
     ('dataframe-overview.html', 'dataframe.html'),
+    ('dataframe-performance.html', 'dataframe-best-practices.html'),
     ('delayed-overview.html', 'delayed.html'),
     ('scheduler-choice.html', 'setup.html'),
     ('diagnostics.html', 'diagnostics-local.html'),


Test patch:
Repository: dask/dask

Issue:
[FEA] Use M methodcaller in melt
### Request
As a dask user, I'd like to be able to potentially use `melt` for multiple types of dataframe data structures. Currently, melt is hard tied to pandas via `pd.melt`. Switching dask's melt to use the `M` methodcaller should make this possible.

Note: I'm going to open a PR for this issue.


Discussion:

Resolving patch:
diff --git a/dask/dataframe/reshape.py b/dask/dataframe/reshape.py
--- a/dask/dataframe/reshape.py
+++ b/dask/dataframe/reshape.py
@@ -8,6 +8,7 @@
 from .utils import (
     is_categorical_dtype, is_scalar, has_known_categories, PANDAS_VERSION
 )
+from ..utils import M
 
 
 ###############################################################
@@ -240,7 +241,7 @@ def melt(frame, id_vars=None, value_vars=None, var_name=None,
 
     from dask.dataframe.core import no_default
 
-    return frame.map_partitions(pd.melt, meta=no_default, id_vars=id_vars,
+    return frame.map_partitions(M.melt, meta=no_default, id_vars=id_vars,
                                 value_vars=value_vars,
                                 var_name=var_name, value_name=value_name,
                                 col_level=col_level, token='melt')


Test patch:
Repository: dask/dask

Issue:
http seek name nloc is not defined with whence=1
**Bug Report***
I've run a trace on an issue with seek into https file.
I get to a point where I have a whence=1, and get *** NameError: name 'nloc' is not defined
self.loc is definitely set
for whence = 1 it's doing
nloc += 

which off hand doesn't make sense, but I could be overlooking something simple.
I also have a self.size so that's good.
attempting it on:
https://six-library.s3.amazonaws.com/sicd_example_RMA_RGZERO_RE16I_IM16I.nitf


Discussion:
Agree, should be `nloc = self.loc + where`.
Would you like to submit the fix as a PR?

Surprised that you are calling this directly, what is your use case.
I'm trying to write an Intake driver.
I was closely trying to reuse code that used local file system.

With local file system I could do:
set_of_open_files = dask.bytes.open_files(urlpath, mode='rb')

Reading basic metadata from the file, then later pulling out a subset of file.
Perhaps not the way to do that, I'm open for suggestions.

I'll give a PR a try (very limited use on my part of github per se)


You are quite right, so you are going beyond typical Dask use here. 
Note that the version of HTTPFile in [fsspec](https://github.com/martindurant/filesystem_spec) doesn't have this problem, and eventually Intake will be moving to use fsspec instead of the functions in Dask (and eventually eventually, Dask should use fsspec).
> I'll give a PR a try (very limited use on my part of github per se)

Thanks @youngsaj! There is documentation about contributing to Dask at http://docs.dask.org/en/latest/develop.html. 
@youngsaj are you still interested in submitting a PR ?  Happy to help/review
(if no reply, I can handle this tomorrow)

Resolving patch:
diff --git a/dask/bytes/http.py b/dask/bytes/http.py
--- a/dask/bytes/http.py
+++ b/dask/bytes/http.py
@@ -146,7 +146,7 @@ def seek(self, where, whence=0):
         if whence == 0:
             nloc = where
         elif whence == 1:
-            nloc += where
+            nloc = self.loc + where
         elif whence == 2:
             nloc = self.size + where
         else:


Test patch:
diff --git a/dask/bytes/tests/test_http.py b/dask/bytes/tests/test_http.py
--- a/dask/bytes/tests/test_http.py
+++ b/dask/bytes/tests/test_http.py
@@ -47,6 +47,22 @@ def test_simple(dir_server):
     assert data == open(os.path.join(dir_server, fn), 'rb').read()
 
 
+def test_loc(dir_server):
+    root = 'http://localhost:8999/'
+    fn = files[0]
+    f = open_files(root + fn)[0]
+    expected = open(os.path.join(dir_server, fn), 'rb').read()
+    with f as f:
+        data = f.read(2)
+        assert data == expected[:2]
+        assert f.loc == 2
+        f.seek(0)
+        data = f.read(3)
+        assert data == expected[:3]
+        f.seek(1, 1)
+        assert f.loc == 4
+
+
 def test_fetch_range_with_headers(dir_server):
     # https://github.com/dask/dask/issues/4479
     root = 'http://localhost:8999/'
Repository: dask/dask

Issue:
Document special handling of dataframe positional arguments to map_partitions.
See #3637.

map_partitions treats dataframes specially when passed as positional arguments.  As far as I can tell, this isn't mentioned in the map_partitions documentation.  I think it should be.



Discussion:
Right, dataframes passed as args first have their partitions aligned, then the function is mapped across all partitions. Agree this would be good to document, can I interest you in submitting a PR?

Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -495,7 +495,8 @@ def map_partitions(self, func, *args, **kwargs):
             Arguments and keywords to pass to the function. The partition will
             be the first argument, and these will be passed *after*. Arguments
             and keywords may contain ``Scalar``, ``Delayed`` or regular
-            python objects.
+            python objects. DataFrame-like args (both dask and pandas) will be
+            repartitioned to align (if necessary) before applying the function.
         $META
 
         Examples
@@ -3775,7 +3776,9 @@ def map_partitions(func, *args, **kwargs):
     args, kwargs :
         Arguments and keywords to pass to the function.  At least one of the
         args should be a Dask.dataframe. Arguments and keywords may contain
-        ``Scalar``, ``Delayed`` or regular python objects.
+        ``Scalar``, ``Delayed`` or regular python objects. DataFrame-like args
+        (both dask and pandas) will be repartitioned to align (if necessary)
+        before applying the function.
     $META
     """
     meta = kwargs.pop('meta', no_default)


Test patch:
Repository: dask/dask

Issue:
da.store is confused when regions is a tuple of tuple of slices.
This works as expected:
```
regions = [(slice1, slice2)]
future = da.store(sources, targets, regions=regions, lock=False, compute=False)
```

While this creates write operations with wrong slices:
```
regions = [(slice1, slice2)]
future = da.store(sources, targets, regions=tuple(regions), lock=False, compute=False)
```
The documentation claims regions can be iterable of tuple of slices, but apparently due to the check in `da.store` for the type of region it only allows non-tuple iterable of tuple of slices as the input.

Either the documentation or the code needs to be changed. I suspect there are other occasions where tuples are treated in special like this.



Discussion:
cc @jakirkham 
AFAICT this has always been true of regions. Please see PR ( https://github.com/dask/dask/pull/1756 ) for details. Specifically it seems to assume that a `tuple` means a single region was specified and that region should be reused for all source/target pairs.

Adding a doc PR that clarifies the existing behavior seems good. If possible improving the error message and checking for this, could also be good. It might be possible to support nesting `tuple`s, but we'd want to think about this a little bit.
Should be a quick fix.
@martindurant does this mean that you plan to execute this fix?

On Tue, Apr 30, 2019 at 11:05 AM Martin Durant <notifications@github.com>
wrote:

> Should be a quick fix.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/4393#issuecomment-488012970>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AACKZTDP5X7ZXF7FOMV7Y43PTBU5VANCNFSM4GQO6JXA>
> .
>


Resolving patch:
diff --git a/dask/array/core.py b/dask/array/core.py
--- a/dask/array/core.py
+++ b/dask/array/core.py
@@ -678,10 +678,12 @@ def store(sources, targets, lock=True, regions=None, compute=True,
         Whether or not to lock the data stores while storing.
         Pass True (lock each file individually), False (don't lock) or a
         particular ``threading.Lock`` object to be shared among all writes.
-    regions: tuple of slices or iterable of tuple of slices
+    regions: tuple of slices or list of tuples of slices
         Each ``region`` tuple in ``regions`` should be such that
         ``target[region].shape = source.shape``
-        for the corresponding source and target in sources and targets, respectively.
+        for the corresponding source and target in sources and targets,
+        respectively. If this is a tuple, the contents will be assumed to be
+        slices, so do not provide a tuple of tuples.
     compute: boolean, optional
         If true compute immediately, return ``dask.delayed.Delayed`` otherwise
     return_stored: boolean, optional


Test patch:
Repository: dask/dask

Issue:
[FEA] Remove pandas dependency of cummin_aggregate and cummax_aggregate
Removing the pandas dependency in the Series/DataFrame vs scalar check within [cummin_aggregate](https://github.com/dask/dask/blob/master/dask/dataframe/methods.py#L147) and [cummax_aggregate](https://github.com/dask/dask/blob/master/dask/dataframe/methods.py#L154) will help allow us to use other backend dataframe libraries such as cuDF for cummin and cummax. When these functions are run in the cummin/cummax operations, the type of argument `x` is determined by the dataframe backend.


Discussion:
@beckernick good idea!  Any interest in submitting a PR ?  You could probably use `is_dataframe_like`

https://github.com/dask/dask/blob/8d13240841fcefb6c2410f94c4d7dbd7f7ea9a66/dask/dataframe/utils.py#L502-L518
@quasiben Yes absolutely! I intended to take this and do exactly that (used that in an update to `_take_last` last week), but I need to prioritize other work until mid-week. Happy to pick this up if nobody else jumps in before that timeline.

Resolving patch:
diff --git a/dask/dataframe/methods.py b/dask/dataframe/methods.py
--- a/dask/dataframe/methods.py
+++ b/dask/dataframe/methods.py
@@ -7,7 +7,7 @@
 from pandas.api.types import is_categorical_dtype, union_categoricals
 from toolz import partition
 
-from .utils import PANDAS_VERSION, is_series_like
+from .utils import PANDAS_VERSION, is_series_like, is_dataframe_like
 from ..utils import Dispatch
 
 if PANDAS_VERSION >= '0.23':
@@ -145,14 +145,14 @@ def describe_aggregate(values):
 
 
 def cummin_aggregate(x, y):
-    if isinstance(x, (pd.Series, pd.DataFrame)):
+    if is_series_like(x) or is_dataframe_like(x):
         return x.where((x < y) | x.isnull(), y, axis=x.ndim - 1)
     else:       # scalar
         return x if x < y else y
 
 
 def cummax_aggregate(x, y):
-    if isinstance(x, (pd.Series, pd.DataFrame)):
+    if is_series_like(x) or is_dataframe_like(x):
         return x.where((x > y) | x.isnull(), y, axis=x.ndim - 1)
     else:       # scalar
         return x if x > y else y


Test patch:
Repository: dask/dask

Issue:
keyerror in dask.config.rename when working with distributed
calling `dask-worker` using `dask==1.2.2`, `distributed==1.28.1`, I get the following error:

`Traceback (most recent call last): File "/opt/conda/envs/worker/bin/dask-worker", line 7, in <module> from distributed.cli.dask_worker import go File "/opt/conda/envs/worker/lib/python3.6/site-packages/distributed/__init__.py", line 3, in <module> from . import config File "/opt/conda/envs/worker/lib/python3.6/site-packages/distributed/config.py", line 55, in <module> dask.config.rename(aliases) File "/opt/conda/envs/worker/lib/python3.6/site-packages/dask/config.py", line 451, in rename del config[k] # TODO: support nested keys KeyError: 'tick-maximum-delay'`

It looks like this was introduced by #4742 when `dask.config.rename` was not properly adjusted, so that it can run into these KeyErrors. Current state of the function:

```def rename(aliases, config=config):
    """ Rename old keys to new keys
    This helps migrate older configuration versions over time
    """
    old = []
    new = {}
    for o, n in aliases.items():
        value = get(o, None, config=config)
        if value is not None:
            old.append(o)
            new[n] = value

    for k in old:
        del config[k]  # TODO: support nested keys

    set(new, config=config)
```

Suggest changing the 3rd to last line to `del config[canonical_name(k, config)]`. Will file an appropriate PR.


Discussion:

Resolving patch:
diff --git a/dask/config.py b/dask/config.py
--- a/dask/config.py
+++ b/dask/config.py
@@ -448,7 +448,7 @@ def rename(aliases, config=config):
             new[n] = value
 
     for k in old:
-        del config[k]  # TODO: support nested keys
+        del config[canonical_name(k, config)]  # TODO: support nested keys
 
     set(new, config=config)
 


Test patch:
diff --git a/dask/tests/test_config.py b/dask/tests/test_config.py
--- a/dask/tests/test_config.py
+++ b/dask/tests/test_config.py
@@ -318,7 +318,7 @@ def test_ensure_file_defaults_to_DASK_CONFIG_directory(tmpdir):
 
 
 def test_rename():
-    aliases = {'foo-bar': 'foo.bar'}
+    aliases = {'foo_bar': 'foo.bar'}
     config = {'foo-bar': 123}
     rename(aliases, config=config)
     assert config == {'foo': {'bar': 123}}