BUG: DataFrame.equals should not care about block order (GH #9330)
State: closed

closes #9330 and another version of the same problem [here on SO](http://stackoverflow.com/questions/29311659/how-can-pandas-dataframes-appear-identical-but-fail-equals) by canonicalizing the block order during an `equals` comparison.

Tested at the block manager level and above it at the frame level.


Labels:
name: API Design
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
lgtm, pls add a release note!

worth adding that test that round trips to hdf (in the example issue)?

Yeah, makes sense.  [I'm hoping that using `dtype.name` sorts out the weird TypeError issue.]  Where should that test go?

put in io/tests/test_pytables.py

yes I think u have to use dtype.name 

the prob is (and maybe want to include a categorical)

is that a categorical dtype is not safe to compare to a numpy dtype (but the name is)

side issue is that if u have 2 cat blocks (cats are block separated, 1 cat per) thrn I sm not sure how they sort

Urf.  Getting the categorical stuff right is going to be a bit tricky. 

Can I assume that NaN will never be an element in a category?  I would have thought it would be disallowed entirely but I don't understand the output when you build one containing one. 

@dsm054 you _can_ have a nan in a category

```
In [5]: df = DataFrame({'A' : Series(list('aabbca')).astype('category',categories=['c','a','b',np.nan]),
   ...:                 'B' : Series(list('aabbca')).astype('category',categories=['c','a','b'])})

In [6]: df
Out[6]: 
   A  B
0  a  a
1  a  a
2  b  b
3  b  b
4  c  c
5  a  a

In [7]: df._data
Out[7]: 
BlockManager
Items: Index([u'A', u'B'], dtype='object')
Axis 1: Int64Index([0, 1, 2, 3, 4, 5], dtype='int64')
CategoricalBlock: slice(0, 1, 1), 1 x 6, dtype: category
CategoricalBlock: slice(1, 2, 1), 1 x 6, dtype: category

```

Blek.  What's the right way to handle putting the categorical blocks -- which won't coalesce even if the categories are the same -- in a canonical order?  

so, I would compare all 'regular' blocks (e.g. non-Categorical/non-Sparse) like you are doing. But these need special handling.

I think you can order by the mgr block order. These map to how the it is represented in the actual DataFrame, and so will be the same. You _could_ actually do this for all blocks I think.

```
In [4]: df._data.blocks[0].mgr_locs.as_array
Out[4]: array([0])

In [5]: df._data.blocks[1].mgr_locs.as_array
Out[5]: array([1])
```

@jreback: okay, I've got a version which passes both my tests and the OP's original case, by sorting on a (block type name, mgr_locs) tuple.  I don't understand how the mgr blocks work well enough to judge that side of things, unfortunately, so I don't know whether matching non-consolidated blocks can be in the wrong order even after this.

Currently passing everywhere except for what looks like an unrelated library build error; probably it'll work the next time it rebuilds.

hmm, can you add a test with 2 categoricals (to your mixed test case)

Beyond `"a:i8;b:category;c:category2;d:category2", # categories`, do you mean?  Two categoricals only?

sorry missed that part of the test

merged via https://github.com/pydata/pandas/commit/e9179feb290a5da3aafbda0efa882afff4415768

thanks @dsm054 
always quality code from you!