Add SparseSeries.to_coo method, a single test and one example.
State: closed

xref #4343
closes #8048

This passes locally run nosetests but is failing on Travis. I think the latest Travis CI changes might have caused failures. I was able to get master to pass on Travis last week but now it is failing. 


Labels:
name: API Design
name: Sparse
description: Sparse Data Type

Comments:
I simplified the test (and removed the .T). Simplifying the test made me realize the implementation was not correct so I have made some changes and added another test for sorted labels and symmetry.

@cottrell need a more robust way to convert. I'll have a look and see if I can help you along here.

I am pushing micro commits to this branch. Please let me know if this is not good github gitiquette and I can squash or save up for a bigger commit. I will hopefully get a few solid blocks of time after Friday, to make scipy_sparse.py less of a hack.

@cottrell not a problem -- we'll ask you to squash at the end before merging anyways.

at the very end you can rebase/squash
until then you can do what u want

I think it is probably ready for some more feedback. I've tried to do the following:
1. cleanup scipy_sparse.py.
2. Add SparseSeries.to_coo to api doc.
3. Add subsection about scipy.sparse in Sparse documentation.
4. Write vbench test.

I am not yet able to run vbench (maybe I should try on python 2.7?). 

I have incorporated your comments and have reorganized the tests. I have also added a from_coo method. I can take the from_coo out if that is just complicating things. The vbench stuff is still untested unless it runs in Travis CI somehow.

@cottrell looking pretty good. pls squash when you have a chance as well.

I had some weird, unreproducible issues with the squashed version not passing Travis CI but it seems to be ok now. For some reason, the SparseSeries.from_coo is not appearing in the API docs while the SparseSeries.to_coo is. 

Small comment: can you check for PEP8? (there are eg some lines that fall of the github screen because of the line length)

@cottrell I added a bunch of mostly documentation comments

I have added a more friendly message when to_coo is called with nlevels<2 (and a test for this Exception) and have incorporated comments from @jorisvandenbossche . In adding a section in the API, I wondered whether it might make sense to someday hook the to_coo method into the Series instance methods simply to aid discovery. It might simply be adding clutter.

Incorporated the comments. I have noticed that pandas.sparse.scipy_sparse is appearing in the public modules (in tab completion IPython). It wasn't obvious to me what the best way of hiding this was.

@cottrell pls add a release note. Make a note under Enhancements. Provide a short example (and provide a link to the new doc section).

Overall looks pretty good, some minor changes.

I think we might need to revisit the conversion algo (at a later time) if perf is an issue, can you post some perf numbers for a couple of cases (I know you have the vbench in there so that is good).

Updated and rebased (there was a conflict in the whatsnew section). Some quick perf numbers here:

http://nbviewer.ipython.org/github/cottrell/notebooks/blob/master/Performance%20scipy.sparse%20examples.ipynb

I forgot to note but I was running on a typical 2.3 Ghz i5 macbook.

Performance seems ok for moderate < 1e7 length indices but definitely could be done better. Could probably be improved along with the MultiIndex groupby operations as these are fairly similar activities.

@cottrell minor change in docs

@jorisvandenbossche @shoyer 

lgtm, any other comments?

Pls add a short example to the doc-string for `to/from_coo` 

Further, I think it might be instructive to have a small doc section that basically shows what this operation is from a reshaping point of view. e.g. using your whatsnew example (not that though this is simple it first densifies so only should do with a relatively small matrix otherwise your memory will blow up; your comprehensive impl is better).

```
In [54]: ss.to_dense().unstack(level=['C','D'])
Out[54]: 
C     a       b    
D     0   1   0   1
A B                
1 1 NaN NaN   1   3
  2   3 NaN NaN NaN
2 1 NaN NaN NaN NaN

In [55]: ss.to_dense().unstack(level=['C','D']).fillna(0).values
Out[55]: 
array([[ 0.,  0.,  1.,  3.],
       [ 3.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.]])

In [56]: ss.to_dense().unstack(level=['C','D']).fillna(0).index.values
Out[56]: array([(1, 1), (1, 2), (2, 1)], dtype=object)

In [57]: ss.to_dense().unstack(level=['C','D']).fillna(0).columns.values
Out[57]: array([('a', 0), ('a', 1), ('b', 0), ('b', 1)], dtype=object)

In [58]: A.todense()
Out[58]: 
matrix([[ 3.,  0.,  0.,  0.],
        [ 0.,  0.,  1.,  3.],
        [ 0.,  0.,  0.,  0.]])
```

I have a strong suspicion you can optimize many of these internal steps by using a MultiIndex. That would speed things up a lot (and simply much of the logic).

Otherwise, the API looks reasonable, though I would use variables with meaningful names like `rows`/`columns` instead of `i`/`j` and return MultiIndexes.

Obviously, performance is not strictly necessary here for this first draft but it's something to think about.

Just to clarify, you don't need to vectorize everything here according to my suggestions. For a first draft, slow is better that not at all.

I will hopefully soon vectorize as I think it will make it more readable. I sort of feel this is whole thing in its cleanest form should really be just a trivial application of a groupby on the levels but I am hitting a problem that I think I hit before which would push me down another path. Does anyone know if the following is a bug? I searched quickly but did not find anything. The problem seems to be only for groupby on index (with levels):

```
In [40]: i = pandas.MultiIndex.from_tuples([(1, 2, 'a', 0),
                                   (1, 2, 'a', 1),
                                   (1, 1, 'b', 0),
                                   (1, 1, 'b', 1),
                                   (2, 1, 'b', 0),
                                   (2, 1, 'b', 1)], names=['a', 'b', 'c', 'd'])

In [41]: a = pandas.Series([0, 1, 2, 3, 4, 5], index=i)

In [42]: a
Out[42]:
a  b  c  d
1  2  a  0    0
         1    1
   1  b  0    2
         1    3
2  1  b  0    4
         1    5
dtype: int64

In [43]: a.groupby(level=['a', 'b'], sort=False).first()
Out[43]:
a  b
1  1    2
   2    0
2  1    4
dtype: int64

In [44]: a.groupby(level=['a', 'b'], sort=True).first()
Out[44]:
a  b
1  1    2
   2    0
2  1    4
dtype: int64

In [45]: a.reset_index().groupby(['a', 'b'], sort=False)[0].first()
Out[45]:
a  b
1  2    0
   1    2
2  1    4
Name: 0, dtype: int64
```

@cottrell can you should how you created `a` in this example? Also, `a.index.levels` and `a.index.labels` would likely demystify this .

@shoyer sorry forgot to include the first bit. Have updated.

Actually, I've posted the sort problem and what seems to fix it here https://github.com/pydata/pandas/issues/9444

I think the problem might occur in a few other places that were not hitting me.

I have incorporated some of the comments from @shoyer which has shortened code quite a bit. This included the patch at https://github.com/pydata/pandas/issues/9444. There are some TODO to review ... worried a bit about relying on the behaviour of some pandas operations for certain steps now (for example dropna on sparse series).

Also, have added examples to the to/from_coo docstrings.

On the downside, the performance is now much, much worse. It looks to me like this is due to the slowness of groupby on MultiIndex levels:

(new version) http://nbviewer.ipython.org/github/cottrell/notebooks/blob/master/pandas%20scipy.sparse%20%28with%20groupby%29%20timiing.ipynb
(old version)
http://nbviewer.ipython.org/github/cottrell/notebooks/blob/master/Performance%20scipy.sparse%20examples.ipynb

I reduced the size of the test matrices in the new version as it was running so slowly.

Performance was too bad using the groupby method so I have reverted to the older python method and left the groupby method commented out (lines 56-57 of scipy_sparse.py).

Some timing results here suggest it is some overhead in the groupby methods that is the difference. 
http://nbviewer.ipython.org/github/cottrell/notebooks/blob/master/pandas%20scipy.sparse%20performance.ipynb

Please let me know if you have any ideas about MultiIndex groupby performance. I have not tried yet, but a diff of pstats output might be useful. 

merged via https://github.com/pydata/pandas/commit/9ac01a7ab02727f69c70401cc4bb8432fe70b83f

@cottrell thank you very much for this!

pls review built docs (give about an hour for travis to build). Pls submit a follow up pr if needed (or for any changes that you think are necessary; or open an issue).

@cottrell There is an error in the docs:

```
>>>-------------------------------------------------------------------------

Exception in /tmp/doc/source/sparse.rst at block ending on line 195

Specify :okexcept: as an option in the ipython:: block to suppress this message

---------------------------------------------------------------------------

ValueError Traceback (most recent call last)

<ipython-input-36-a2b3f3f12d26> in <module>()

1 A, rows, columns = ss.to_coo(row_levels=['A', 'B', 'C'],

2 column_levels=['D'],

----> 3 sort_labels=False)

/home/travis/build/pydata/pandas/pandas/sparse/series.pyc in to_coo(self, row_levels, column_levels, sort_labels)

709 """

710 A, rows, columns = _sparse_series_to_coo(

--> 711 self, row_levels, column_levels, sort_labels=sort_labels)

712 return A, rows, columns

713

/home/travis/build/pydata/pandas/pandas/sparse/scipy_sparse.pyc in _sparse_series_to_coo(ss, row_levels, column_levels, sort_labels)

113

114 v, i, j, rows, columns = _to_ijv(

--> 115 ss, row_levels=row_levels, column_levels=column_levels, sort_labels=sort_labels)

116 sparse_matrix = scipy.sparse.coo_matrix(

117 (v, (i, j)), shape=(len(rows), len(columns)))

/home/travis/build/pydata/pandas/pandas/sparse/scipy_sparse.pyc in _to_ijv(ss, row_levels, column_levels, sort_labels)

90

91 i_coord, i_labels = get_indexers(row_levels)

---> 92 j_coord, j_labels = get_indexers(column_levels)

93

94 return values, i_coord, j_coord, i_labels, j_labels

/home/travis/build/pydata/pandas/pandas/sparse/scipy_sparse.pyc in get_indexers(levels)

80

81 labels_to_i = _get_index_subset_to_coord_dict(

---> 82 ss.index, levels, sort_labels=sort_labels)

83 #######################################################################

84 #######################################################################

/home/travis/build/pydata/pandas/pandas/sparse/scipy_sparse.pyc in _get_index_subset_to_coord_dict(index, subset, sort_labels)

74 ilabels, sort_labels=sort_labels)

75 labels_to_i = Series(labels_to_i)

---> 76 labels_to_i.index = MultiIndex.from_tuples(labels_to_i.index)

77 labels_to_i.index.names = [index.names[i] for i in subset]

78 labels_to_i.name = 'value'

/home/travis/build/pydata/pandas/pandas/core/index.pyc in from_tuples(cls, tuples, sortorder, names)

3604 tuples = tuples.values

3605

-> 3606 arrays = list(lib.tuples_to_object_array(tuples).T)

3607 elif isinstance(tuples, list):

3608 arrays = list(lib.to_object_array_tuples(tuples).T)

/home/travis/build/pydata/pandas/pandas/lib.so in pandas.lib.tuples_to_object_array (pandas/lib.c:54298)()

ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'

<<<-------------------------------------------------------------------------
```

@jorisvandenbossche

Yes, this is hopefully addressed here:

https://github.com/pydata/pandas/pull/9583

I was building docs only with python 3 so didn't catch this on my setup.