internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: dask/dask

Issue:
da.stack dtype
Greetings dask dudes,

I'm using `dask` version `0.14.1`

I stumbled across the following behaviour with respect to the `dtype` of a **computed** `stack`, and I was curious as to whether it is as expected:

First, create some data ...

```python
import dask.array as da                           
import numpy as np                 

# integer
i = np.array(np.arange(3), dtype='i4')
# float
f = np.array(np.arange(3), dtype='f4')

di = da.from_array(i, chunks=i.shape)
df = da.from_array(f, chunks=f.shape)
```

Now, `stack` the arrays in alternate orders ...
```python
didf = da.stack([di, df], axis=0)
dfdi = da.stack([df, di], axis=0)
```
And show the results, paying attention to the lazy `dtype` and the computed `dtype` ...
```python
print(didf)
print(dfdi)
print(didf.compute().dtype)
print(dfdi.compute().dtype)
```
Yields ...
```python
dask.array<stack, shape=(2, 3), dtype=float64, chunksize=(1, 3)>
dask.array<stack, shape=(2, 3), dtype=float64, chunksize=(1, 3)>
int32
float32
```
Comparing this with `concatenate`, we see a different result, but a result I would expect ...
```python
didf = da.concatenate([di, df], axis=0)
dfdi = da.concatenate([df, di], axis=0)
print(didf)
print(dfdi)
print(didf.compute().dtype)
print(dfdi.compute().dtype)
```
Yields ...
```python
dask.array<concatenate, shape=(6,), dtype=float64, chunksize=(3,)>
dask.array<concatenate, shape=(6,), dtype=float64, chunksize=(3,)>
float64
float64
```
Is this non-symmetric `dtype` behaviour of `stack` as you would expect?

Many thanks!



Discussion:

Resolving patch:
diff --git a/dask/array/core.py b/dask/array/core.py
--- a/dask/array/core.py
+++ b/dask/array/core.py
@@ -2255,6 +2255,9 @@ def stack(seq, axis=0):
     uc_args = list(concat((x, ind) for x in seq))
     _, seq = unify_chunks(*uc_args)
 
+    dt = reduce(np.promote_types, [a.dtype for a in seq])
+    seq = [x.astype(dt) for x in seq]
+
     assert len(set(a.chunks for a in seq)) == 1  # same chunks
     chunks = (seq[0].chunks[:axis] + ((1,) * n,) + seq[0].chunks[axis:])
 
@@ -2271,8 +2274,6 @@ def stack(seq, axis=0):
     dsk = dict(zip(keys, values))
     dsk2 = sharedict.merge((name, dsk), *[a.dask for a in seq])
 
-    dt = reduce(np.promote_types, [a.dtype for a in seq])
-
     return Array(dsk2, name, chunks, dtype=dt)
 
 


Test patch:
diff --git a/dask/array/tests/test_array_core.py b/dask/array/tests/test_array_core.py
--- a/dask/array/tests/test_array_core.py
+++ b/dask/array/tests/test_array_core.py
@@ -277,6 +277,15 @@ def test_stack_scalars():
     assert s.compute().tolist() == [np.arange(4).mean(), np.arange(4).sum()]
 
 
+def test_stack_promote_type():
+    i = np.arange(10, dtype='i4')
+    f = np.arange(10, dtype='f4')
+    di = da.from_array(i, chunks=5)
+    df = da.from_array(f, chunks=5)
+    res = da.stack([di, df])
+    assert_eq(res, np.stack([i, f]))
+
+
 @pytest.mark.skipif(LooseVersion(np.__version__) < '1.10.0',
                     reason="NumPy doesn't yet support stack")
 def test_stack_rechunk():
Repository: dask/dask

Issue:
COMPAT: pandas 0.20.0
https://github.com/pandas-dev/pandas/pull/16223

we moved the imports for ``pandas.util.hashing`` to ``pandas.util``. So I will put up a PR to reflect this change for dask. (these are deprecated in the original location).


Discussion:

Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -10,7 +10,6 @@
 
 from toolz import merge, first, unique, partition_all, remove
 import pandas as pd
-from pandas.util.decorators import cache_readonly
 import numpy as np
 
 try:
@@ -38,8 +37,10 @@
 no_default = '__no_default__'
 
 if PANDAS_VERSION >= '0.20.0':
+    from pandas.util import cache_readonly
     pd.core.computation.expressions.set_use_numexpr(False)
 else:
+    from pandas.util.decorators import cache_readonly
     pd.computation.expressions.set_use_numexpr(False)
 
 
diff --git a/dask/dataframe/hashing.py b/dask/dataframe/hashing.py
--- a/dask/dataframe/hashing.py
+++ b/dask/dataframe/hashing.py
@@ -25,7 +25,7 @@
 # versioneer, since the release tags aren't ancestors of master. As such, we
 # need to use this hacky awfulness to check if we're > 0.19.2.
 if PANDAS_VERSION >= '0.20.0':
-    from pandas.util.hashing import hash_pandas_object
+    from pandas.util import hash_pandas_object
 elif PANDAS_VERSION not in ('0.19.1', '0.19.2') and PANDAS_VERSION > '0.19.0+460':
     from pandas.tools.hashing import hash_pandas_object
 else:


Test patch:
Repository: dask/dask

Issue:
Fancy indexing of arrays with length equal to one
I have an application that needs to take the first and the last element of an array.
If the array has length 1 it should return an array with the only element twice.
This works in numpy but, when i do it with dask.arrays i get the following error:

```python
>>> import numpy as np
>>> import dask
>>> a = np.zeros(1)
>>> a[[0,-1]]
array([ 0.,  0.])
>>> d = dask.array.from_array(a, (1,))
>>> d[[0,-1]]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "dask\array\core.py", line 1243, in __getitem__
    dsk, chunks = slice_array(out, self.name, self.chunks, index)
  File "dask\array\slicing.py", line 139, in slice_array
    dsk_out, bd_out = slice_with_newaxes(out_name, in_name, blockdims, index)
  File "dask\array\slicing.py", line 161, in slice_with_newaxes
    dsk, blockdims2 = slice_wrap_lists(out_name, in_name, blockdims, index2)
  File "dask\array\slicing.py", line 202, in slice_wrap_lists
    check_index(i, sum(bd))
  File "dask\array\slicing.py", line 713, in check_index
    raise IndexError("Index out of bounds %s" % dimension)
IndexError: Index out of bounds 1
```


Discussion:

Resolving patch:
diff --git a/dask/array/slicing.py b/dask/array/slicing.py
--- a/dask/array/slicing.py
+++ b/dask/array/slicing.py
@@ -709,7 +709,7 @@ def check_index(ind, dimension):
     """
     if isinstance(ind, list):
         x = np.array(ind)
-        if (x >= dimension).any() or (x <= -dimension).any():
+        if (x >= dimension).any() or (x < -dimension).any():
             raise IndexError("Index out of bounds %s" % dimension)
     elif isinstance(ind, slice):
         return


Test patch:
diff --git a/dask/array/tests/test_slicing.py b/dask/array/tests/test_slicing.py
--- a/dask/array/tests/test_slicing.py
+++ b/dask/array/tests/test_slicing.py
@@ -549,3 +549,10 @@ def test_None_overlap_int():
 
 def test_negative_n_slicing():
     assert_eq(da.ones(2, chunks=2)[-2], np.ones(2)[-2])
+
+
+def test_negative_list_slicing():
+    x = np.arange(5)
+    dx = da.from_array(x, chunks=2)
+    assert_eq(dx[[0, -5]], x[[0, -5]])
+    assert_eq(dx[[4, -1]], x[[4, -1]])
Repository: dask/dask

Issue:
Complex conjugate generates unstable hash
When running this script, the dask `name` changes on every run:
```python
import numpy as np
import dask.array as da

A = np.ones((10, 10))*5
B = np.ones((10, 10))*6
A = da.from_array(A, chunks=A.shape)
B = da.from_array(B, chunks=B.shape)
C = A + 1j * B
print(C.conj().name)  # this changes on every run
```
Shouldn't this be stable? [Here](https://github.com/dask/dask/issues/2228)'s a related issue which has already been solved.

Thanks!


Discussion:
Hmmm, this is weird. It looks like numpy ufuncs don't have stable pickle representations. It's consistent per-run, but not consistent on each initialization of python. Issue is true for any ufunc:

```python
(dask) jcrist dask $ cat test.py
import numpy as np
import pickle

print(pickle.dumps(np.sin))
(dask) jcrist dask $ python test.py
b'\x80\x03cnumpy.core\n_ufunc_reconstruct\nq\x00X\x10\x00\x00\x00numpy.core.umathq\x01X\x03\x00\x00\x00sinq\x02\x86q\x03Rq\x04.'
(dask) jcrist dask $ python test.py
b'\x80\x03cnumpy.core\n_ufunc_reconstruct\nq\x00X\n\x00\x00\x00numpy.coreq\x01X\x03\x00\x00\x00sinq\x02\x86q\x03Rq\x04.'
```

Looking at the numpy code, this is because numpy uses the `pickle.whichmodule` function to find the module with the object, which iterates through the (non-deterministic) `sys.modules.items()` to find the first module with the object.

Fixing now.

Resolving patch:
diff --git a/dask/base.py b/dask/base.py
--- a/dask/base.py
+++ b/dask/base.py
@@ -401,6 +401,15 @@ def normalize_array(x):
     normalize_token.register(np.dtype, repr)
     normalize_token.register(np.generic, repr)
 
+    @normalize_token.register(np.ufunc)
+    def normalize_ufunc(x):
+        try:
+            name = x.__name__
+            if getattr(np, name) is x:
+                return 'np.' + name
+        except:
+            return normalize_function(x)
+
 
 def tokenize(*args, **kwargs):
     """ Deterministic token


Test patch:
diff --git a/dask/tests/test_base.py b/dask/tests/test_base.py
--- a/dask/tests/test_base.py
+++ b/dask/tests/test_base.py
@@ -155,6 +155,17 @@ def test_tokenize_numpy_memmap_no_filename():
         assert tokenize(b) == tokenize(b)
 
 
+@pytest.mark.skipif('not np')
+def test_tokenize_numpy_ufunc_consistent():
+    assert tokenize(np.sin) == '02106e2c67daf452fb480d264e0dac21'
+    assert tokenize(np.cos) == 'c99e52e912e4379882a9a4b387957a0b'
+
+    # Make a ufunc that isn't in the numpy namespace. Similar to
+    # any found in other packages.
+    inc = np.frompyfunc(lambda x: x + 1, 1, 1)
+    assert tokenize(inc) == tokenize(inc)
+
+
 def test_normalize_base():
     for i in [1, 1.1, '1', slice(1, 2, 3)]:
         assert normalize_token(i) is i
Repository: dask/dask

Issue:
Make to_parquet robust to empty partitions
to_parquet currently throws a ValueError with` raise ValueError("All partitions were empty")` if  all partitions are empty.

Ideally it should be able to cope with empty partitions as for some workflows there is no way of telling if partitions will be empty ahead of time.

to_csv is an example where it copes with empty partitions currently.


Discussion:

Resolving patch:
diff --git a/dask/dataframe/io/parquet.py b/dask/dataframe/io/parquet.py
--- a/dask/dataframe/io/parquet.py
+++ b/dask/dataframe/io/parquet.py
@@ -121,6 +121,11 @@ def _read_fastparquet(fs, paths, myopen, columns=None, filters=None,
     if isinstance(divisions[0], np.datetime64):
         divisions = [pd.Timestamp(d) for d in divisions]
 
+    if not dsk:
+        # empty dataframe
+        dsk = {(name, 0): meta}
+        divisions = (None, None)
+
     return out_type(dsk, name, meta, divisions)
 
 
@@ -455,8 +460,6 @@ def _write_metadata(writes, filenames, fmd, path, metadata_fn, myopen, sep):
                     chunk.file_path = fn
                 fmd.row_groups.append(rg)
 
-    if len(fmd.row_groups) == 0:
-        raise ValueError("All partitions were empty")
     fastparquet.writer.write_common_metadata(metadata_fn, fmd, open_with=myopen,
                                              no_row_groups=False)
 


Test patch:
diff --git a/dask/dataframe/io/tests/test_parquet.py b/dask/dataframe/io/tests/test_parquet.py
--- a/dask/dataframe/io/tests/test_parquet.py
+++ b/dask/dataframe/io/tests/test_parquet.py
@@ -374,10 +374,6 @@ def test_empty_partition(fn):
     assert_eq(ddf2.compute(), ddf3.compute(), check_names=False,
               check_index=False)
 
-    ddf2 = ddf[ddf.a <= -5]
-    with pytest.raises(ValueError):
-        ddf2.to_parquet(fn)
-
 
 def test_timestamp_index():
     with tmpfile() as fn:
@@ -465,3 +461,14 @@ def test_to_parquet_lazy(tmpdir, get):
     ddf2 = dd.read_parquet(tmpdir)
 
     assert_eq(ddf, ddf2)
+
+
+def test_empty(fn):
+    df = pd.DataFrame({'a': ['a', 'b', 'b'], 'b': [4, 5, 6]})[0:0]
+    ddf = dd.from_pandas(df, npartitions=2)
+    ddf.to_parquet(fn, write_index=False)
+    files = os.listdir(fn)
+    assert '_metadata' in files
+    out = dd.read_parquet(fn).compute()
+    assert len(out) == 0
+    assert_eq(out, df)
Repository: dask/dask

Issue:
bag.to_dataframe() not working with topk()
import bag from dask as db
db.from_sequence([(i,i**2) for i in range(5)]).topk(3, lambda x: x[1]).to_dataframe().compute()
>>> ValueError: DataFrame constructor not properly called!

CPython 3.6.1
IPython 6.1.0
dask.__version__ = 0.15.0


Discussion:
It seems to occur here:
```
> /Users/mdurant/Downloads/dask/dask/bag/core.py(1992)to_dataframe()
   1989
   1990 def to_dataframe(seq, columns, dtypes):
   1991     import pandas as pd
-> 1992     res = pd.DataFrame(reify(seq), columns=list(columns))
   1993     return res.astype(dtypes, copy=False)
```

where wrapping `reify` (which produces a tuple) in `list` solves the problem. I don't know if that is the right solution.

Resolving patch:
diff --git a/dask/bag/core.py b/dask/bag/core.py
--- a/dask/bag/core.py
+++ b/dask/bag/core.py
@@ -1989,5 +1989,9 @@ def split(seq, n):
 
 def to_dataframe(seq, columns, dtypes):
     import pandas as pd
-    res = pd.DataFrame(reify(seq), columns=list(columns))
+    seq = reify(seq)
+    # pd.DataFrame expects lists, only copy if necessary
+    if not isinstance(seq, list):
+        seq = list(seq)
+    res = pd.DataFrame(seq, columns=list(columns))
     return res.astype(dtypes, copy=False)


Test patch:
diff --git a/dask/bag/tests/test_bag.py b/dask/bag/tests/test_bag.py
--- a/dask/bag/tests/test_bag.py
+++ b/dask/bag/tests/test_bag.py
@@ -823,12 +823,13 @@ def check_parts(df, sol):
     dd.utils.assert_eq(df, sol, check_index=False)
     check_parts(df, sol)
 
-    # Works with iterators
-    b = db.from_sequence(range(100), npartitions=5).map_partitions(iter)
+    # Works with iterators and tuples
     sol = pd.DataFrame({'a': range(100)})
-    df = b.to_dataframe(columns=sol)
-    dd.utils.assert_eq(df, sol, check_index=False)
-    check_parts(df, sol)
+    b = db.from_sequence(range(100), npartitions=5)
+    for f in [iter, tuple]:
+        df = b.map_partitions(f).to_dataframe(columns=sol)
+        dd.utils.assert_eq(df, sol, check_index=False)
+        check_parts(df, sol)
 
 
 ext_open = [('gz', GzipFile), ('', open)]
Repository: dask/dask

Issue:
Computing read_parquet with filters can return empty list
If you provide a filter that selects no partitions, `compute()` will return an empty list instead of an empty DataFrame with correct columns and dtypes.

Maybe not relevant, but plain and easy to fix in the same context problem - poor documentation for `filters` argument. It never says that the filters are applied only to partitions. It may be easily mistaken with `where` in `pandas.HDFStore.select`.


Discussion:

Resolving patch:
diff --git a/dask/dataframe/io/parquet.py b/dask/dataframe/io/parquet.py
--- a/dask/dataframe/io/parquet.py
+++ b/dask/dataframe/io/parquet.py
@@ -237,7 +237,10 @@ def read_parquet(path, columns=None, filters=None, categories=None, index=None,
     columns: list or None
         List of column names to load
     filters: list
-        List of filters to apply, like ``[('x', '>' 0), ...]``
+        List of filters to apply, like ``[('x', '>' 0), ...]``. This implements
+        row-group (partition) -level filtering only, i.e., to prevent the
+        loading of some chunks of the data, and only if relevant statistics
+        have been included in the metadata.
     index: string or None (default) or False
         Name of index column to use if that column is sorted;
         False to force dask to not use any column as the index


Test patch:
Repository: dask/dask

Issue:
ResourceProfile has different behaviour using context manager vs `register`
I'm using the dask profilers with the pattern
```
for profiler in profilers: profiler.register()
foo() # Do work, some of it dask, some of it other
for profiler in profilers: profiler.unregister()
```
(although with some try/finally stuff to make it more robust). I had assumed that this would be equivalent to running `foo` under a context manager per profiler, except that the `with` statement doesn't allow for unpacking so can't take a variable number of context managers (and `contextlib.nested` is deprecated and I'm on Python 2 so can't use `ExitStack`). However, the `ResourceProfiler` code has specific checks for whether it is currently being used as a context manager (`_enter` member) and if not, it stops tracking resources as soon as a computation is complete (in `_finish`), which leads to gaps in the graph whenever no dask computation is running.

Example code:
```python
#!/usr/bin/env python
import dask.array as da
from dask.diagnostics import ResourceProfiler
import time

def foo():
    x = da.ones(1000000000, chunks=1000000)
    da.dot(x, x).compute()  # Keep things busy for a while
    time.sleep(3)
    da.dot(x, x).compute()  # Keep things busy again

profiler = ResourceProfiler(dt=0.2)
profiler.register()
foo()
profiler.unregister()
profiler.visualize()
```


Discussion:
> except that the with statement doesn't allow for unpacking so can't take a variable number of context managers

For non variable numbers, you can do:

```
with ResourceProfiler() as rprof, Profiler( )as prof:
    ...
```

which is my common pattern.

> it stops tracking resources as soon as a computation is complete (in _finish)

This mirrors the other profilers, which also only track during dask computation. As a context manager the `ResourceProfiler` is actually an exception, as it can be used to profile non-dask code (always tracks around a context). I'm slightly against changing this behavior. What would you want this to be instead?
I'd got the impression from the documentation that they were [intended to be] exactly equivalent, with the context manager just being a convenient way to ensure cleanup, but it sounds like that wasn't the intention. Maybe it just needs a bit of clarification in the documentation.

Resolving patch:
diff --git a/dask/diagnostics/profile.py b/dask/diagnostics/profile.py
--- a/dask/diagnostics/profile.py
+++ b/dask/diagnostics/profile.py
@@ -134,6 +134,10 @@ class ResourceProfiler(Callback):
     manually.
 
     >>> prof.clear()  # doctest: +SKIP
+
+    Note that when used as a context manager data will be collected throughout
+    the duration of the enclosed block. In contrast, when registered globally
+    data will only be collected while a dask scheduler is active.
     """
     def __init__(self, dt=1):
         self._tracker = _Tracker(dt)


Test patch:
Repository: dask/dask

Issue:
Support schema for sql_read_table
Hi,

I tried to use the dask read_sql_table function with a table within a schema, but it returns 'NoSuchTableError'. 

Here are more details:
My database is hosted by amazon rds with postegres

If I try to connect using psycopg2, it works:
```
import psycopg2
import dask.dataframe as dd
url_db = 'postgres://...'
connection = psycopg2.connect(url_db)

cur = connection.cursor()
cur.execute(
    """SELECT * FROM my_schema.my_table""")
print(cur.fetchall())
# this works
```

So the connection is right, but when I try to use dask:

`df = dd.read_sql(table='my_table', schema='my_schema', uri=url_db, index_col='id')`

then I got:

```
---------------------------------------------------------------------------
NoSuchTableError                          Traceback (most recent call last)
<ipython-input-44-e6602fc736cb> in <module>()
...
```

Furthermore, if I try to load a table not in a schema, it works:
`df = dd.read_sql(table='another_table', uri=url_db, index_col='id')`

then it also works...

So I wonder if sql schema is supported in dask ? I didn't found anything on the web about it.



Discussion:

Resolving patch:
diff --git a/dask/dataframe/io/sql.py b/dask/dataframe/io/sql.py
--- a/dask/dataframe/io/sql.py
+++ b/dask/dataframe/io/sql.py
@@ -68,7 +68,9 @@ def read_sql_table(table, uri, index_col, divisions=None, npartitions=None,
     engine = sa.create_engine(uri)
     meta = sa.MetaData()
     if isinstance(table, six.string_types):
-        table = sa.Table(table, meta, autoload=True, autoload_with=engine)
+        schema = kwargs.pop('schema', None)
+        table = sa.Table(table, meta, autoload=True, autoload_with=engine,
+                         schema=schema)
 
     index = (table.columns[index_col] if isinstance(index_col, six.string_types)
              else index_col)


Test patch:
Repository: dask/dask

Issue:
String Accessors in Converted DataFrame Columns
Whenever I try to access strings in a dataframe which are previously converted to strings (ie. datall[['A']] = datall[['A']].applymap(str) and then datall['A']=datall['A'].str[:5]) I get a TypeError: 'StringAccessor' object has no attribute '__getitem__'.

This is reproducible as follows:

```
import pandas as pd
import dask.dataframe as dd
import numpy as np

def float_apply(x):
    try:
        return float(x)
    except ValueError:
        return float('nan')

def string_apply(x):
    try:
        return str(x)
    except ValueError:
        return str('nan')

df = pd.DataFrame(np.random.random_integers(0,6,size=(20, 6)), columns=list('ABCDEF'))

data = dd.from_pandas(df, npartitions = 2)

data=data.applymap(float_apply)

data[['A']] = data[['A']].applymap(string_apply)
data['A'] = data['A'].str[:1]

print data.compute()

```

This will work with pandas dataframes. If .compute() is run prior to this then it works fine but that is probably suboptimal for large datasets.


Discussion:

Resolving patch:
diff --git a/dask/dataframe/accessor.py b/dask/dataframe/accessor.py
--- a/dask/dataframe/accessor.py
+++ b/dask/dataframe/accessor.py
@@ -115,3 +115,12 @@ def _validate(self, series):
     @derived_from(pd.core.strings.StringMethods)
     def split(self, pat=None, n=-1):
         return self._function_map('split', pat=pat, n=n)
+
+    def __getitem__(self, index):
+        return self._series.map_partitions(str_get, index,
+                                           meta=self._series._meta)
+
+
+def str_get(series, index):
+    """ Implements series.str[index] """
+    return series.str[index]


Test patch:
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -1339,7 +1339,7 @@ def test_datetime_accessor():
 
 
 def test_str_accessor():
-    df = pd.DataFrame({'x': ['a', 'b', 'c', 'D'], 'y': [1, 2, 3, 4]},
+    df = pd.DataFrame({'x': ['abc', 'bcd', 'cdef', 'DEFG'], 'y': [1, 2, 3, 4]},
                       index=['e', 'f', 'g', 'H'])
 
     a = dd.from_pandas(df, 2, sort=False)
@@ -1376,6 +1376,9 @@ def test_str_accessor():
         assert_eq(a.x.str.contains('a', regex=regex), df.x.str.contains('a', regex=regex))
         assert set(a.x.str.contains('a', regex=regex).dask) == set(a.x.str.contains('a', regex=regex).dask)
 
+    assert_eq(df.x.str[:2], df.x.str[:2])
+    assert_eq(a.x.str[1], a.x.str[1])
+
 
 def test_empty_max():
     meta = make_meta({'x': 'i8'})