Repository: pandas-dev/pandas

Issue:
Read and write spss data format
It would be nice to be able to import spss dant with read_spss and export it using to_spss.



Discussion:
The following package might be useful https://www.ibm.com/developerworks/community/files/app?lang=en#/person/270003ERMT/file/b77a0da0-2f47-454b-b505-5404b242d78c

Does SPSS offer no export formats compatible with pandas? Or, Is there relevant
data contained in the proprietary file format which can't otherwise be accessed?

This may still make sense if for example users without access to SPSS frequently 
get SPSS files (AKA. the microsoft word problem). Not sure if that's the case though.

Actually, I do not use SPSS. But I have some SPSS data files that I want to explore. I may work with R but the data is so huge that this is not possible. The only way I did find is to tideous. I have to use PSPP to prepare the subset and the import it to R. With all the back and forth to add some variables and to master the SPSS syntax. Since pandas can deal with huge datasets, I do think it should provide import from SPSS. And I am willing to test it.  

That was a good enough reason for stata support, yeah.

Note that this particular [package](https://pypi.python.org/pypi/savReaderWriter/) weighs in at 90-180MB, includes a large chunk of binary
proprietary (yet free to use)  code and the pypi version doesn't work on linux 64bit (though the bug 
was fixed two months ago on bitbucket), so It's maturity is slightly suspect.

Basic usage doesn't require any explicit pandas cooperation:

```
import savReaderWriter as s
df= pd.DataFrame(list(s.SavReader('foo.sav')))
```

is all it takes. 

There's something to be said for pandas accepting data rather then data formats.
If there's a package that reads format X and produces data, then pandas implicitly 
supports format X.

Definitely worth a FAQ entry though, I'm sure other users have this need. 
Care to write some prose and make a PR?

Thank you very much for looking through  this problem.
I tried to import some .sav data into a pandas dataframe as you did but I ended having the following error on a win64:
WindowsError: [Error 193] %1 is not a valid Win32 application in Python 

Sorry @y-p This seems to be a reported error. I am using a 64bit python on a 64bit machine which is the case that is problematic according to this discussion https://bitbucket.org/fomcl/savreaderwriter/issue/12/win64-error 

Hi All, 
any progress on this topic?  
I was searching a lot about this and dont found any answer so I use this small code to import sav's to pandas  but only works in Python 3.5 

```
import pandas as pd
import numpy as np
import savWriterReader as spss

with spss.SavReaderNp ("some_sav_file.sav") as reader:
    records = reader.all()
df = pd.DataFrame(records)
df.head()
```

Hope to work to someone. 

Have a great day!
Lots of data is made available in SPSS so this tool would be very useful, especially for social scientists and economists. If the solution of @ThinkOnData works, it seems it should be an easy improvement. I will try it out and may submit a PR.
I second the usefulness of `read_spss` and `to_spss`. I am currently entering a collaboration with a team of Information Systems scholars that use SPSS and UI-based Structural Equation Modeling tools. 

Using SPSS to manage the master survey data seems to be a common approach (cf., [Gaskin, 2016](https://www.youtube.com/watch?v=AS7ZxOZgU4c)). 

As the first step in introducing a computational approach to the collaboration, I am writing a script that preprocesses the survey data we have collected. Being able to easily read and write SPSS would certainly be helpful here.
For those interested in this issue, I think contributing better pandas support (or suggesting it) to the https://bitbucket.org/fomcl/savreaderwriter would be a good first step. 
If the package would have direct support to read a file into a DataFrame, and advertise this, I think it would already help a lot of people without needing to directly add it to pandas itself.

SPSS is the only file format that is exportable from common survey tools like Qualtrics, SurveyGizmo, and SurveyMonkey that allows you to preserve both the values and the labels for many variables. 

Survey data seems to always be represented as one row per response and one column per questions or question choice. For single select questions or Open ended , the values are usually coded as a single number per choice, so a column may have a 1, 2, or 3 for male, female, or 'prefer not to state'. If you can multi-select, most survey tools export questions like Q2_1, Q2_2, Q2_3... for each of the possible choices, and then each cell has a 1 for Selected and a null/SYSMIS/NaN value if it was not selected. Sometimes, that missing data is also coded as a -99 or other values. 

Finally, SPSS has 2 properties, VARIABLE LABELS and VALUE LABELS that contain the Strings that tend to correspond to question text/choice text. 

If you export a survey file as CSV from any of those tools, you are presented with the choice to either take the strings from each questions (e.g. "Strongly Agree" will be what is in each cell where that was selected) or, you can have a '5'. the trouble is that for many types of analysis you want both. In pandas, I think it would be easy to treat all of these as category labels with pd.categorical. 

The common use case for this is to see what the average scale rating is for a question - e.g. in a Disagree <---> Agree scale is to get a mean response / st dev. But, you may also want to produce a cross tab that shows you count/percent for each category column. SPSS can do this pretty well, but, it is expensive, slow, and has a really lame internal syntax language. 

If pandas had full support for SPSS files, and could write them out, it would be very helpful for doing both initial data exploration, question aggregation, data restructuring, text analysis, and then write out the new file to leverage other downstream tools like Wincross for reporting/analysis that require SPSS files and are easy for business/non-techncial people to use. 

_As someone that deals with a lot of survey data, I'm happy to talk/chat/answer any questions I can about this, and to test anything out in SPSS if it would help anyone. Note that I'm a novice when it comes to python/pandas, but I've been using SPSS for a long time and am looking to move away from it completely._ 
I have written a wrapper for the C library Readstat named pyreadstat which reads SPSS sav, zsav and por files: github.com/Roche/pyreadstat
It would be great if this functionality was available directly from Pandas, e.g. via `read_spss`. @TomAugspurger @jreback @jorisvandenbossche (sorry for the explicit mentions, I don't know how to at the whole dev team) would this be an option (given that this requires a C lib)? @ofajardo would you be willing to merge your code?
It’s more likely that we would have an optional dependency on that package that a read_spss would use. Similar to what we do with pyarrow and parquet. 

> On May 22, 2019, at 04:20, Clemens Brunner <notifications@github.com> wrote:
> 
> It would be great if this functionality was available directly from Pandas, e.g. via read_spss. @TomAugspurger @jreback @jorisvandenbossche (sorry for the explicit mentions, I don't know how to at the whole dev team) would this be an option (given that this requires a C lib)? @ofajardo would you be willing to merge your code?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.

@cbrnr @TomAugspurger I am willing to help and contribute
Great! So this could be as simple as adding `pandas/io/spss.py` and wrapping relevant functions of `pyreadstat` (and making sure to import it only within functions and not globally). Since only a data frame should be returned, some effort is probably be necessary in using the meta information for creating suitable column names, data types, and so on.

Also,`pyreadstat` can also read SAS and Stata files, which Pandas already supports natively (but `pyreadstat` is much faster). I don't know how you would like to handle these file formats (ignore for now, create separate modules/functions or integrate with existing readers), but I think for now just adding support for SPSS files would be a good plan.
> Also,pyreadstat can also read SAS and Stata files, which Pandas already supports natively (but pyreadstat is much faster). I don't know how you would like to handle these file formats (ignore for now, 

I think ignore for now, but we can certainly revisit once we have spss taken care of. Once there's interest we can add an `engine` keyword to `read_sas` and `read_stata`.

Resolving patch:
diff --git a/pandas/__init__.py b/pandas/__init__.py
--- a/pandas/__init__.py
+++ b/pandas/__init__.py
@@ -105,7 +105,7 @@
 
     # misc
     read_clipboard, read_parquet, read_feather, read_gbq,
-    read_html, read_json, read_stata, read_sas)
+    read_html, read_json, read_stata, read_sas, read_spss)
 
 from pandas.util._tester import test
 import pandas.testing
diff --git a/pandas/io/api.py b/pandas/io/api.py
--- a/pandas/io/api.py
+++ b/pandas/io/api.py
@@ -16,5 +16,6 @@
 from pandas.io.pickle import read_pickle, to_pickle
 from pandas.io.pytables import HDFStore, read_hdf
 from pandas.io.sas import read_sas
+from pandas.io.spss import read_spss
 from pandas.io.sql import read_sql, read_sql_query, read_sql_table
 from pandas.io.stata import read_stata
diff --git a/pandas/io/spss.py b/pandas/io/spss.py
new file mode 100644
--- /dev/null
+++ b/pandas/io/spss.py
@@ -0,0 +1,41 @@
+from pathlib import Path
+from typing import Optional, Sequence, Union
+
+from pandas.compat._optional import import_optional_dependency
+
+from pandas.api.types import is_list_like
+from pandas.core.api import DataFrame
+
+
+def read_spss(path: Union[str, Path],
+              usecols: Optional[Sequence[str]] = None,
+              convert_categoricals: bool = True) -> DataFrame:
+    """
+    Load an SPSS file from the file path, returning a DataFrame.
+
+    .. versionadded 0.25.0
+
+    Parameters
+    ----------
+    path : string or Path
+        File path
+    usecols : list-like, optional
+        Return a subset of the columns. If None, return all columns.
+    convert_categoricals : bool, default is True
+        Convert categorical columns into pd.Categorical.
+
+    Returns
+    -------
+    DataFrame
+    """
+    pyreadstat = import_optional_dependency("pyreadstat")
+
+    if usecols is not None:
+        if not is_list_like(usecols):
+            raise TypeError("usecols must be list-like.")
+        else:
+            usecols = list(usecols)  # pyreadstat requires a list
+
+    df, _ = pyreadstat.read_sav(path, usecols=usecols,
+                                apply_value_formats=convert_categoricals)
+    return df


Test patch:
diff --git a/pandas/tests/api/test_api.py b/pandas/tests/api/test_api.py
--- a/pandas/tests/api/test_api.py
+++ b/pandas/tests/api/test_api.py
@@ -81,7 +81,7 @@ class TestPDApi(Base):
                   'read_gbq', 'read_hdf', 'read_html', 'read_json',
                   'read_msgpack', 'read_pickle', 'read_sas', 'read_sql',
                   'read_sql_query', 'read_sql_table', 'read_stata',
-                  'read_table', 'read_feather', 'read_parquet']
+                  'read_table', 'read_feather', 'read_parquet', 'read_spss']
 
     # top-level to_* funcs
     funcs_to = ['to_datetime', 'to_msgpack',
diff --git a/pandas/tests/io/data/labelled-num-na.sav b/pandas/tests/io/data/labelled-num-na.sav
new file mode 100755
Binary files /dev/null and b/pandas/tests/io/data/labelled-num-na.sav differ
diff --git a/pandas/tests/io/data/labelled-num.sav b/pandas/tests/io/data/labelled-num.sav
new file mode 100755
Binary files /dev/null and b/pandas/tests/io/data/labelled-num.sav differ
diff --git a/pandas/tests/io/data/labelled-str.sav b/pandas/tests/io/data/labelled-str.sav
new file mode 100755
Binary files /dev/null and b/pandas/tests/io/data/labelled-str.sav differ
diff --git a/pandas/tests/io/data/umlauts.sav b/pandas/tests/io/data/umlauts.sav
new file mode 100755
Binary files /dev/null and b/pandas/tests/io/data/umlauts.sav differ
diff --git a/pandas/tests/io/test_spss.py b/pandas/tests/io/test_spss.py
new file mode 100644
--- /dev/null
+++ b/pandas/tests/io/test_spss.py
@@ -0,0 +1,74 @@
+import numpy as np
+import pytest
+
+import pandas as pd
+from pandas.util import testing as tm
+
+pyreadstat = pytest.importorskip("pyreadstat")
+
+
+def test_spss_labelled_num(datapath):
+    # test file from the Haven project (https://haven.tidyverse.org/)
+    fname = datapath("io", "data", "labelled-num.sav")
+
+    df = pd.read_spss(fname, convert_categoricals=True)
+    expected = pd.DataFrame({"VAR00002": "This is one"}, index=[0])
+    expected["VAR00002"] = pd.Categorical(expected["VAR00002"])
+    tm.assert_frame_equal(df, expected)
+
+    df = pd.read_spss(fname, convert_categoricals=False)
+    expected = pd.DataFrame({"VAR00002": 1.0}, index=[0])
+    tm.assert_frame_equal(df, expected)
+
+
+def test_spss_labelled_num_na(datapath):
+    # test file from the Haven project (https://haven.tidyverse.org/)
+    fname = datapath("io", "data", "labelled-num-na.sav")
+
+    df = pd.read_spss(fname, convert_categoricals=True)
+    expected = pd.DataFrame({"VAR00002": ["This is one", None]})
+    expected["VAR00002"] = pd.Categorical(expected["VAR00002"])
+    tm.assert_frame_equal(df, expected)
+
+    df = pd.read_spss(fname, convert_categoricals=False)
+    expected = pd.DataFrame({"VAR00002": [1.0, np.nan]})
+    tm.assert_frame_equal(df, expected)
+
+
+def test_spss_labelled_str(datapath):
+    # test file from the Haven project (https://haven.tidyverse.org/)
+    fname = datapath("io", "data", "labelled-str.sav")
+
+    df = pd.read_spss(fname, convert_categoricals=True)
+    expected = pd.DataFrame({"gender": ["Male", "Female"]})
+    expected["gender"] = pd.Categorical(expected["gender"])
+    tm.assert_frame_equal(df, expected)
+
+    df = pd.read_spss(fname, convert_categoricals=False)
+    expected = pd.DataFrame({"gender": ["M", "F"]})
+    tm.assert_frame_equal(df, expected)
+
+
+def test_spss_umlauts(datapath):
+    # test file from the Haven project (https://haven.tidyverse.org/)
+    fname = datapath("io", "data", "umlauts.sav")
+
+    df = pd.read_spss(fname, convert_categoricals=True)
+    expected = pd.DataFrame({"var1": ["the ä umlaut",
+                                      "the ü umlaut",
+                                      "the ä umlaut",
+                                      "the ö umlaut"]})
+    expected["var1"] = pd.Categorical(expected["var1"])
+    tm.assert_frame_equal(df, expected)
+
+    df = pd.read_spss(fname, convert_categoricals=False)
+    expected = pd.DataFrame({"var1": [1.0, 2.0, 1.0, 3.0]})
+    tm.assert_frame_equal(df, expected)
+
+
+def test_spss_usecols(datapath):
+    # usecols must be list-like
+    fname = datapath("io", "data", "labelled-num.sav")
+
+    with pytest.raises(TypeError, match="usecols must be list-like."):
+        pd.read_spss(fname, usecols="VAR00002")