Defer Series.str.get_dummies to pandas.get_dummies
State: closed

- [x] closes #19618
- [x] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Enhancement
name: Categorical
description: Categorical Data Type

Comments:
@jreback the test are failing due to an import but I didn't change this file:
```
pandas/core/reshape/reshape.py:17: in <module>
        from pandas.core.arrays import SparseArray
E   ImportError: cannot import name 'SparseArray'
The command "ci/run_tests.sh" exited with 4.
```

Do you know why this is?

Possibly a circular import. Can you `import pandas` locally?

On Thu, Jun 6, 2019 at 4:37 AM Josh Levy-Kramer <notifications@github.com>
wrote:

> @jreback <https://github.com/jreback> the test are failing due to an
> import but I didn't change this file:
>
> pandas/core/reshape/reshape.py:17: in <module>
>         from pandas.core.arrays import SparseArray
> E   ImportError: cannot import name 'SparseArray'
> The command "ci/run_tests.sh" exited with 4.
>
> Do you know why this is?
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/26686?email_source=notifications&email_token=AAKAOIVQMBROVH2WU2HIIBDPZDLFLA5CNFSM4HU64FC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXCJTNA#issuecomment-499423668>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIW6GQISYMAT5CXSX3LPZDLFLANCNFSM4HU64FCQ>
> .
>

Thanks I think your right - it's a circular import.

> Can you `import pandas` locally?

Do you mean try `import pandas.core.reshape.reshape as reshape` and then use `reshape.get_dummies` as this does not work?

Whats the best say to resolve this. Should I move the function to another python file?


# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26686?src=pr&el=h1) Report
> Merging [#26686](https://codecov.io/gh/pandas-dev/pandas/pull/26686?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/891a419a5155e6b42c0696a81cf853b6f3febbf7?src=pr&el=desc) will **decrease** coverage by `49.94%`.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26686/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26686?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #26686       +/-   ##
===========================================
- Coverage   91.87%   41.93%   -49.95%     
===========================================
  Files         174      174               
  Lines       50661    50697       +36     
===========================================
- Hits        46547    21260    -25287     
- Misses       4114    29437    +25323
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `41.93% <0%> (+0.06%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26686?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/strings.py](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3RyaW5ncy5weQ==) | `38.07% <0%> (-60.85%)` | :arrow_down: |
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `0% <0%> (-99.37%)` | :arrow_down: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `0% <0%> (-91.16%)` | :arrow_down: |
| [pandas/io/sas/sas\_xport.py](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=) | `0% <0%> (-90.1%)` | :arrow_down: |
| [pandas/core/sparse/scipy\_sparse.py](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL3NjaXB5X3NwYXJzZS5weQ==) | `10.14% <0%> (-89.86%)` | :arrow_down: |
| ... and [129 more](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26686?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26686?src=pr&el=footer). Last update [891a419...70e077a](https://codecov.io/gh/pandas-dev/pandas/pull/26686?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26686?src=pr&el=h1) Report
> Merging [#26686](https://codecov.io/gh/pandas-dev/pandas/pull/26686?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/891a419a5155e6b42c0696a81cf853b6f3febbf7?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26686/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26686?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26686      +/-   ##
==========================================
- Coverage   91.87%   91.87%   -0.01%     
==========================================
  Files         174      179       +5     
  Lines       50661    50689      +28     
==========================================
+ Hits        46547    46569      +22     
- Misses       4114     4120       +6
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.46% <100%> (+0.05%)` | :arrow_up: |
| #single | `41.1% <100%> (-0.77%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26686?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/strings.py](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3RyaW5ncy5weQ==) | `98.92% <100%> (ø)` | :arrow_up: |
| [pandas/tseries/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/feather\_format.py](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2ZlYXRoZXJfZm9ybWF0LnB5) | `25% <0%> (-64.48%)` | :arrow_down: |
| [pandas/io/parquet.py](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnF1ZXQucHk=) | `65.59% <0%> (-18.88%)` | :arrow_down: |
| [pandas/core/computation/check.py](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vY2hlY2sucHk=) | `85.71% <0%> (-6.6%)` | :arrow_down: |
| [pandas/core/config\_init.py](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29uZmlnX2luaXQucHk=) | `92.91% <0%> (-3.94%)` | :arrow_down: |
| [pandas/compat/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9fX2luaXRfXy5weQ==) | `92% <0%> (-1.55%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-0.59%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.88% <0%> (-0.24%)` | :arrow_down: |
| [pandas/core/arrays/integer.py](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2ludGVnZXIucHk=) | `96.3% <0%> (-0.05%)` | :arrow_down: |
| ... and [62 more](https://codecov.io/gh/pandas-dev/pandas/pull/26686/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26686?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26686?src=pr&el=footer). Last update [891a419...a98aa26](https://codecov.io/gh/pandas-dev/pandas/pull/26686?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@TomAugspurger I resolved the issue by locally importing. I hope thats ok
> How does the performance of this compare to the original?

```
s = pd.Series([*['a|b|asdf|s|d|d', np.nan, 'a|c']*10000])

%%timeit
s.str.get_dummies()
```

Original version:
`155 ms ± 13.9 ms per loop`

New:
`55.9 ms ± 3.94 ms`
> How is the performance of this approach compared to master? Do we have a benchmark for it in `asv_bench`?

There is an existing benchmark `strings.Dummies`:
```bash
$ cd asv_bench
$ asv continuous upstream/master HEAD -b strings.Dummies
$ asv compare upstream/master HEAD

All benchmarks:

       before           after         ratio
     [891a419a]       [70e077ab]
     <master>         <defer_str_dummies_to_get_dummies>
       3.81±0.04s        7.47±0.7s    ~1.96  strings.Dummies.time_get_dummies
```

It shows that the new function takes about twice as long - which I suppose isn't great. However, the small test I did before (see previous comment) showed a speed increase. So maybe it depends on the test
Can you do a bit of research / profiling on when and why the new
implementation is slower?

On Thu, Jun 27, 2019 at 8:42 AM Josh Levy-Kramer <notifications@github.com>
wrote:

> How is the performance of this approach compared to master? Do we have a
> benchmark for it in asv_bench?
>
> There is an existing benchmark strings.Dummies:
>
> $ asv continuous upstream/master HEAD -s -b strings.Dummies
>
> $ asv compare upstream/master HEAD
>
>
>
> All benchmarks:
>
>
>
>        before           after         ratio
>
>      [891a419a]       [70e077ab]
>
>      <master>         <defer_str_dummies_to_get_dummies>
>
>        3.81±0.04s        7.47±0.7s    ~1.96  strings.Dummies.time_get_dummies
>
> It shows that the new function takes about twice as long - which I suppose
> isn't great. However, the small test I did before (see previous comment)
> showed a speed increase. So maybe it depends on the test
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/26686?email_source=notifications&email_token=AAKAOIXMFFBAEM6TVYK4ZSTP4S7VVA5CNFSM4HU64FC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYXE62Q#issuecomment-506351466>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIRW4GI7UA2R7PZZYXLP4S7VVANCNFSM4HU64FCQ>
> .
>

@joshlk can you address merge conflict and question above?
closing as stale, if you'd like to continue pls ping to reopen.
Not sure if I'll have time to look into this, but chiming in to mention I came across this while facing the same issue. Implementing a pure-python text-rank for short text that exploits CSR matrix for document-term (based on [this](http://blog.samuelmh.com/2015/02/pagerank-sparse-matrices-python-ipython.html), and originally hoped to use the `str.get_dummies()` function since it accepts separator arguments. But I ran into the same memory error with `int64`. 

For now having to fall back on scikit-learn for document-term matrix computation, but the get_dummies function is so straight-forward/streamlined for short text (e.g. twitter) that interfacing with sparse arrays and networkx is really nice...no need for full-on sklearn dependencies in small utilities, etc. 

If I do get some time to mess with this I'll see what I can do, but in the mean time just a vote for how convenient merging the two `get_dummies()` behaviors would be.