PERF: Improve performance of StataReader's processing of missing values
State: closed

Improve performance of StataReader when converting columns
with missing values

xref #25772

- [x] closes #N/A
- [x] tests added / passed N/A, perf only
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Performance
description: Memory or execution speed performance
name: IO Stata
description: read_stata, to_stata

Comments:
can you add an asv that covers this case?
Hello @bashtage! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-03-20 13:16:12 UTC
can you merge master 
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25780?src=pr&el=h1) Report
> Merging [#25780](https://codecov.io/gh/pandas-dev/pandas/pull/25780?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/8e54e556a6083c3ef26f246db36fb0ac556e846f?src=pr&el=desc) will **decrease** coverage by `49.51%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25780/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25780?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #25780       +/-   ##
===========================================
- Coverage   91.26%   41.74%   -49.52%     
===========================================
  Files         172      172               
  Lines       52965    52965               
===========================================
- Hits        48337    22109    -26228     
- Misses       4628    30856    +26228
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `41.74% <ø> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25780?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/25780/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/25780/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/25780/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/25780/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/25780/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `0% <0%> (-99.36%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/25780/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `0% <0%> (-95.46%)` | :arrow_down: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/25780/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `0% <0%> (-91.17%)` | :arrow_down: |
| [pandas/io/sas/sas\_xport.py](https://codecov.io/gh/pandas-dev/pandas/pull/25780/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=) | `0% <0%> (-90.15%)` | :arrow_down: |
| [pandas/core/tools/numeric.py](https://codecov.io/gh/pandas-dev/pandas/pull/25780/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdG9vbHMvbnVtZXJpYy5weQ==) | `10.44% <0%> (-89.56%)` | :arrow_down: |
| [pandas/io/json/normalize.py](https://codecov.io/gh/pandas-dev/pandas/pull/25780/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vbm9ybWFsaXplLnB5) | `8.16% <0%> (-88.78%)` | :arrow_down: |
| ... and [129 more](https://codecov.io/gh/pandas-dev/pandas/pull/25780/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25780?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25780?src=pr&el=footer). Last update [8e54e55...2eda26b](https://codecov.io/gh/pandas-dev/pandas/pull/25780?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25780?src=pr&el=h1) Report
> Merging [#25780](https://codecov.io/gh/pandas-dev/pandas/pull/25780?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/37d04a3d3df82c3abb80afc2ef6fcfa0b7d5df44?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25780/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25780?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #25780   +/-   ##
=======================================
  Coverage   91.27%   91.27%           
=======================================
  Files         173      173           
  Lines       53002    53002           
=======================================
  Hits        48375    48375           
  Misses       4627     4627
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.83% <ø> (ø)` | :arrow_up: |
| #single | `41.77% <ø> (ø)` | :arrow_up: |


------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25780?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25780?src=pr&el=footer). Last update [37d04a3...e5f3c06](https://codecov.io/gh/pandas-dev/pandas/pull/25780?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

lgtm. ping on green.
@jreback I think the failures are unrelated to my changes.
master is passing, so maybe you have an older build. pls merge master and let's see.
@jreback Green.
thanks @bashtage does this fully close the issue?
No,  there is still an issue with duplicate valuelables which can't be handeled by categories which require unique labels.  There are 2 solutions to that: 1. mung the value label and preserve the data (an integer) or 2. assign all values with the same label to be the same.  1 is probably better since there isn't much loss of fidelity (can just find the munged names and fix, plus the underlying integer values are still correct for the labeled categories).   But this needs more thought and work.