Add reader for SPSS (.sav) files
State: closed

- [x] closes #5768 (at least the reading part, this PR does not cover writing SPSS files)
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

I haven't added a test yet because I wanted to ask which test .sav file I should use (and where to put it). Also, there's no whatsnew entry yet - in which section should this entry go (and I assume it will be out for 0.25.0, so I'll have to change 0.24.3 to 0.25.0 in the docstring).

This PR adds the capability to load SPSS .sav files with `df = pd.io.read_spss("spss_file.sav")`. Currently, there are two optional arguments: `usecols` (should be self-explanatory, let me know if you don't want me to handle a simple `str`) and `categorical`, which maps to the `apply_value_formats` parameter in `read_sav`. With `categorical=True`, a categorical columns is created with the labels from the .sav file. If `False`, numbers will be used.

A few open questions:
- Which additional optional arguments should be made available? [Pyreadstat](https://github.com/Roche/pyreadstat/blob/master/pyreadstat/pyreadstat.pyx#L208-L244) has `dates_as_pandas_datetime`, `encoding`, and `user_missing` which I haven't mapped yet.
- Should the function be called `read_spss` or `read_sav`? SPSS files have the extension `sav`, but the R haven package has a function `read_spss` (which is why I'd prefer `read_spss`).
- Are there any additional meta information bits that could be used/integrated into the data frame? `pyreadstat.read_sav` returns a dataframe and meta-information separately, which I think we shouldn't do in pandas.

Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: IO Stata
description: read_stata, to_stata

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26537?src=pr&el=h1) Report
> Merging [#26537](https://codecov.io/gh/pandas-dev/pandas/pull/26537?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/0a516c15dcb4b3b6eae8bc231a5c8656acfca5b4?src=pr&el=desc) will **decrease** coverage by `50.06%`.
> The diff coverage is `25%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26537/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26537?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #26537       +/-   ##
===========================================
- Coverage   91.76%    41.7%   -50.07%     
===========================================
  Files         174      175        +1     
  Lines       50629    50637        +8     
===========================================
- Hits        46462    21119    -25343     
- Misses       4167    29518    +25351
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `41.7% <25%> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26537?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/api.py](https://codecov.io/gh/pandas-dev/pandas/pull/26537/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2FwaS5weQ==) | `100% <100%> (ø)` | :arrow_up: |
| [pandas/io/spss.py](https://codecov.io/gh/pandas-dev/pandas/pull/26537/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nwc3MucHk=) | `14.28% <14.28%> (ø)` | |
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/26537/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/26537/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/26537/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/26537/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/26537/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/26537/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `0% <0%> (-99.37%)` | :arrow_down: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/26537/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `0% <0%> (-91.16%)` | :arrow_down: |
| [pandas/io/sas/sas\_xport.py](https://codecov.io/gh/pandas-dev/pandas/pull/26537/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=) | `0% <0%> (-90.1%)` | :arrow_down: |
| ... and [131 more](https://codecov.io/gh/pandas-dev/pandas/pull/26537/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26537?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26537?src=pr&el=footer). Last update [0a516c1...0407cc5](https://codecov.io/gh/pandas-dev/pandas/pull/26537?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26537?src=pr&el=h1) Report
> Merging [#26537](https://codecov.io/gh/pandas-dev/pandas/pull/26537?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/430f0fd04646a0ab44ccbfbad706dfe22e9fabfe?src=pr&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `50%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26537/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26537?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26537      +/-   ##
==========================================
- Coverage   91.88%   91.86%   -0.03%     
==========================================
  Files         179      180       +1     
  Lines       50696    50710      +14     
==========================================
+ Hits        46581    46583       +2     
- Misses       4115     4127      +12
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.45% <50%> (-0.02%)` | :arrow_down: |
| #single | `41.1% <50%> (-0.08%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26537?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/api.py](https://codecov.io/gh/pandas-dev/pandas/pull/26537/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2FwaS5weQ==) | `100% <100%> (ø)` | :arrow_up: |
| [pandas/io/spss.py](https://codecov.io/gh/pandas-dev/pandas/pull/26537/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nwc3MucHk=) | `46.15% <46.15%> (ø)` | |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26537/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26537/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.88% <0%> (-0.12%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26537/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `90.84% <0%> (-0.11%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26537?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26537?src=pr&el=footer). Last update [430f0fd...e55b8c4](https://codecov.io/gh/pandas-dev/pandas/pull/26537?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

I've added some test files from the [haven](https://github.com/tidyverse/haven/tree/master/tests/testthat) package - how do we attribute the original authors? They don't have a standard license, so maybe we have to ask them for permission to use their test files?

One test file containing dates cannot be loaded because `pyreadstat` produces an error - this is reported in https://github.com/Roche/pyreadstat/issues/26.

I've updated `install.rst` and I'm currently trying to see if a Travis job successfully completes when `pyreadstat` is installed.
> I've added some test files from the haven package - how do we attribute the original authors? They don't have a standard license, so maybe we have to ask them for permission to use their test files?

Seems like haven has an MIT license, according to https://cran.r-project.org/web/packages/haven/index.html. Does that sound right @hadley?

If so, I think you can include include Haven's license file in our licenses folder, and we'll be good. It'd be good to note the source of these tests files in `test_spss.py`.
Yeah, that's fine with me. (I generally follow the US and consider data to be un-copyrightable, although in this case I guess it's the specific form that it's important not so much the data)
All tests pass now ~~(currently re-running because of a wrong imports order)~~. Two open questions:

1. I have currently added `pyreadstat` only to one Travis job (3.7). Where would you like this dependency to be added?
2. The Haven license consists of two files ([MIT](https://cran.r-project.org/web/licenses/MIT) and [LICENSE](https://cran.r-project.org/web/packages/haven/LICENSE)). How and where should these files be added (e.g. how should we rename these files)?
Hello @cbrnr! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:











There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-06-15 07:07:22 UTC
@jreback I've implemented all changes.
Yes, `pyreadstat` is only available via `pip` and not via `conda`.
Fixed the formatting issues, hopefully this will come back green and then it should be ready to merge.
@WillAyd could you please elaborate?
Not sure what happened to my comment but was asking to change `Union[str, Sequence[str], None]` to `Optional[Union[str, Sequence[str]]`
@WillAyd done!
@cbrnr Pyreadstat is available both with pip and conda. In the README that is explained in the How to install section
I didn't see that you have conda-forge added, sorry about that. I've addressed all comments.
No idea what's going on with Azure, could someone please restart?
can you merge master (you have a conflict)
OK, I've rebased.
looks good. @jorisvandenbossche @TomAugspurger @bashtage ok with this?
The only comment I would make is based on helping the Stata reader: are the SAV files comprehensive in the sense that they contain every combination of data storage features? This was an issue with the Stata reader where the original test files did not contain all features which lead to a slow but stead set of error reports, until finally a set was created that had every possible storage format and encoding. 


I've taken the test files from the Haven R package, which uses the same underlying C library [readstat](https://github.com/WizardMac/ReadStat) to load SAV files. One of these files contains a strange timestamp (year 201416), which currently produces an error when read with pyreadstat (but readstat as well as Haven can read this file). @ofajardo has already fixed this by adding a new parameter in https://github.com/Roche/pyreadstat/issues/26. Once he makes a new release, I can add a new parameter `disable_datetime_conversion ` to `pd.read_spss` (then we need to require at least this version). Alternatively, I could add `kwargs` which get passed to the underlying pyreadstat function (and we don't check anything on the pandas side).

There's also a second test file I didn't use from the Haven package, because it contains ~~value~~variable labels which currently cannot be converted to something pandas offers. The file can be read successfully though.

Regarding the second test file: SAV files can have column names and column labels. pyreadstat can read both, but we have to set either one of the two as `df.columns`. There is a parameter that controls which gets used, but I assume pandas doesn't have a place within the `DataFrame` object to store both column names and column labels?
@cbrnr regarding the second test file, which variable labels cannot be converted to pandas? I think pyreadstat is getting everything that the C Readstat offers, but you can have either the original value or the labelled value but not both (similar situation as with  column labels), this can be handled passing keyword arguments to pyreadstat.
In any case tests in pyreadstat do check for every possible feature in the spss files that the C library can cover, at least to the best of my knowledge, so it should be relatively safe. 
Yes, that's what I meant - you cannot have both since a DataFrame presumably cannot store both entities (or can it?).
@jreback @jorisvandenbossche @TomAugspurger @bashtage do you want me to make any additional changes?
@TomAugspurger I've addressed your comments.
Rebased and fixed comments, this should be good to go.
Some CI failures
https://travis-ci.org/pandas-dev/pandas/jobs/543229555#L2502

On Sat, Jun 8, 2019 at 3:29 PM Clemens Brunner <notifications@github.com>
wrote:

> Rebased and fixed comments, this should be good to go.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/26537?email_source=notifications&email_token=AAKAOIQNM2BDUQ2MYTCKVHLPZQJD3A5CNFSM4HP3GQF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXH5PJA#issuecomment-500160420>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIX6JEVW3XAIOZXTAR3PZQJD3ANCNFSM4HP3GQFQ>
> .
>

Fixed.
Hopefully fixed now.