Repository: dask/dask

Issue:
dataframe column names getting scrambled
I am loading from data from bcolz into a dask dataframe. After applying a `.where()` operation, the column names get scrambled.

Unfortunately I haven't been able to reproduce this with a simpler example. But here it is. Quite stuck and frustrated. (Dask version 0.7.3.)

``` python
from dask import dataframe as dd
bdf = dd.from_bcolz(fname)
bdf.loc[0].compute()
```

gives

```
npart    3073441.000000
time           0.000000
vort           0.000001
x            225.023438
y            -19.984375
Name: 0, dtype: float32
```

but if I do

```
bdf.where(bdf.time==0).loc[0].compute()
```

I get

```
npart    3073441.000000
time           0.000000
x              0.000001
y            225.023438
vort         -19.984375
Name: 0, dtype: float32
```

The `x` and `vort` columns have been exchanged.

Any idea what is wrong and how to fix it?



Discussion:
Hrm, odd.  Do you have a small example that you can share that I can take a
look at?

What happens if you use dd.from_array?

bc = bcolz.ctable(rootdir='...')
df = dd.from_array(bc)

On Fri, Oct 16, 2015 at 1:10 PM, Ryan Abernathey notifications@github.com
wrote:

> I am loading from data from bcolz into a dask dataframe. After applying a
> .where() operation, the column names get scrambled.
> 
> Unfortunately I haven't been able to reproduce this with a simpler
> example. But here it is. Quite stuck and frustrated. (Dask version 0.7.3.)
> 
> from dask import dataframe as dd
> bdf = dd.from_bcolz(fname)
> bdf.loc[0].compute()
> 
> gives
> 
> npart    3073441.000000
> time           0.000000
> vort           0.000001
> x            225.023438
> y            -19.984375
> Name: 0, dtype: float32
> 
> but if I do
> 
> bdf.where(bdf.time==0).loc[0].compute()
> 
> I get
> 
> npart    3073441.000000
> time           0.000000
> x              0.000001
> y            225.023438
> vort         -19.984375
> Name: 0, dtype: float32
> 
> The x and vort columns have been exchanged.
> 
> Any idea what is wrong and how to fix it?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/blaze/dask/issues/787.

I am struggling to create a small example (will try), but in the meantime, I can confirm that your version with `dd.from_array` does NOT produce the same problem. (Also pure pandas does not produce the problem.)

The `from_bcolz` function will be a more efficient because it'll leverage
the column-store nature of bcolz.

I find myself tempted to remove it though, just to reduce the maintenance
burden of dask.dataframe.  I'm not surprised to find a bug here.  This
particular codepath is infrequently used.

However, from_bcolz gets used then we should definitely continue to support
it.  E-mail responses like this help to motivate this work.

On Fri, Oct 16, 2015 at 2:03 PM, Ryan Abernathey notifications@github.com
wrote:

> I am struggling to create a small example (will try), but in the meantime,
> I can confirm that your version with dd.from_array does NOT produce the
> same problem. (Also pure pandas does not make produce the problem.)
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/blaze/dask/issues/787#issuecomment-148836414.

As you might be able to infer from the issues I have been leaving around, I am trying to build a pipeline for storing and processing a ton of binary data. I read your [post on castra](http://matthewrocklin.com/blog/work/2015/08/28/Storage/) and that sounded like a perfect solution for my problem. Unfortunately I couldn't easily get castra working (see ContinuumIO/anaconda-issues#456), so I decided to fall back on bcolz, which seems to share some of the advantages.

At this point, I have a bunch of data in bcolz format. So I would be happy if dask would continue to support it directly. But I realize these tools are quite new and still in flux.

OK.  I'll take a look at this issue tomorrow.  It's probably just something
silly.

On Mon, Oct 19, 2015 at 6:28 PM, Ryan Abernathey notifications@github.com
wrote:

> As you might be able to infer from the issues I have been leaving around,
> I am trying to build a pipeline for storing and processing a ton of binary
> data. I read your post on castra
> http://matthewrocklin.com/blog/work/2015/08/28/Storage/ and that
> sounded like a perfect solution for my problem. Unfortunately I couldn't
> easily get castra working (see anaconda-issues/issues#456), so I decided to
> fall back on bcolz, which seems to share some of the advantages.
> 
> At this point, I have a bunch of data in bcolz format. So I would be happy
> if dask would continue to support it directly. But I realize these tools
> are quite new and still in flux.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/blaze/dask/issues/787#issuecomment-149393109.


Resolving patch:
diff --git a/dask/dataframe/io.py b/dask/dataframe/io.py
--- a/dask/dataframe/io.py
+++ b/dask/dataframe/io.py
@@ -395,8 +395,8 @@ def from_bcolz(x, chunksize=None, categorize=True, index=None, **kwargs):
     if categorize:
         for name in x.names:
             if (np.issubdtype(x.dtype[name], np.string_) or
-                    np.issubdtype(x.dtype[name], np.unicode_) or
-                    np.issubdtype(x.dtype[name], np.object_)):
+                np.issubdtype(x.dtype[name], np.unicode_) or
+                np.issubdtype(x.dtype[name], np.object_)):
                 a = da.from_array(x[name], chunks=(chunksize * len(x.names),))
                 categories[name] = da.unique(a)
 
@@ -415,7 +415,7 @@ def from_bcolz(x, chunksize=None, categorize=True, index=None, **kwargs):
                 (dataframe_from_ctable,
                  x,
                  (slice(i * chunksize, (i + 1) * chunksize),),
-                 None, categories))
+                 columns, categories))
                for i in range(0, int(ceil(len(x) / chunksize))))
 
     result = DataFrame(dsk, new_name, columns, divisions)
@@ -459,20 +459,23 @@ def dataframe_from_ctable(x, slc, columns=None, categories=None):
 
     """
     import bcolz
-    if columns is not None:
-        if isinstance(columns, tuple):
-            columns = list(columns)
-        x = x[columns]
+    if columns is None:
+        columns = x.dtype.names
+    if isinstance(columns, tuple):
+        columns = list(columns)
+
+    x = x[columns]
 
     if isinstance(x, bcolz.ctable):
-        chunks = [x[name][slc] for name in x.names]
+        chunks = [x[name][slc] for name in columns]
         if categories is not None:
             chunks = [pd.Categorical.from_codes(np.searchsorted(categories[name],
                                                                 chunk),
                                                 categories[name], True)
                        if name in categories else chunk
-                       for name, chunk in zip(x.names, chunks)]
-        return pd.DataFrame(dict(zip(x.names, chunks)))
+                       for name, chunk in zip(columns, chunks)]
+        return pd.DataFrame(dict(zip(columns, chunks)), columns=columns)
+
     elif isinstance(x, bcolz.carray):
         chunk = x[slc]
         if categories is not None and columns and columns in categories:


Test patch:
diff --git a/dask/dataframe/tests/test_io.py b/dask/dataframe/tests/test_io.py
--- a/dask/dataframe/tests/test_io.py
+++ b/dask/dataframe/tests/test_io.py
@@ -228,6 +228,15 @@ def test_from_bcolz_filename():
         assert list(d.x.compute()) == [1, 2, 3]
 
 
+def test_from_bcolz_column_order():
+    bcolz = pytest.importorskip('bcolz')
+
+    t = bcolz.ctable([[1, 2, 3], [1., 2., 3.], ['a', 'b', 'a']],
+                     names=['x', 'y', 'a'])
+    df = dd.from_bcolz(t, chunksize=2)
+    assert list(df.loc[0].compute().index) == ['x', 'y', 'a']
+
+
 def test_skipinitialspace():
     text = normalize_text("""
     name, amount