Repository: pandas-dev/pandas
Pull request: BUG: Fix problem with SparseDataFrame not persisting to csv
- [x] closes #19384
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
SparseDataFrame to_csv returns IndexError: too many indices for array
Calling to_csv on a SparseDataFrame returns IndexError: too many indices for array.
```python
import pandas as pd
from scipy.sparse import rand

x = rand(100, 10001, density=0.2, format='csr')
df = pd.SparseDataFrame(x)
df.to_csv("tmp.csv")
```

Calling .to_dense() on the df prior to writing out, of course, works fine.
```python
df.to_dense().to_csv("tmp.csv")
```

I'd like to be able to save sparse data to disk.  If for some reason it shouldn't be written out this way, is there any way to add that in the doc or the error message?

#### Expected Output

Nontype, no exception, and a csv on the file system.

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Darwin
OS-release: 16.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.21.0
pytest: None
pip: 9.0.1
setuptools: 28.8.0
Cython: None
numpy: 1.13.3
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.1.0
psycopg2: 2.7.3.2 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>

Hints from issue discussion:

Commit message: BUG: Fix problem with SparseDataFrame not persisting to csv

Patch to review:
diff --git a/doc/source/whatsnew/v0.23.0.txt b/doc/source/whatsnew/v0.23.0.txt
index 1890636bc8e1a..93bcc90a2a514 100644
--- a/doc/source/whatsnew/v0.23.0.txt
+++ b/doc/source/whatsnew/v0.23.0.txt
@@ -497,7 +497,7 @@ I/O
 - Bug in :func:`DataFrame.to_parquet` where an exception was raised if the write destination is S3 (:issue:`19134`)
 - :class:`Interval` now supported in :func:`DataFrame.to_excel` for all Excel file types (:issue:`19242`)
 - :class:`Timedelta` now supported in :func:`DataFrame.to_excel` for xls file type (:issue:`19242`, :issue:`9155`)
--
+- Bug in :class:`SparseDataFrame.to_csv` where too many indices for values (:issue:`19384`)
 
 Plotting
 ^^^^^^^^
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
index c2d3d0852384c..2e26d91a50b81 100644
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -704,7 +704,9 @@ def to_native_types(self, slicer=None, na_rep='nan', quoting=None,
                         **kwargs):
         """ convert to our native types format, slicing if desired """
 
-        values = self.values
+        # values = self.values
+        values = self.get_values()
+
         if slicer is not None:
             values = values[:, slicer]
         mask = isna(values)
diff --git a/pandas/tests/io/formats/test_to_csv.py b/pandas/tests/io/formats/test_to_csv.py
index e12a7196dce6b..da6a5b0676ec4 100644
--- a/pandas/tests/io/formats/test_to_csv.py
+++ b/pandas/tests/io/formats/test_to_csv.py
@@ -253,3 +253,12 @@ def test_to_csv_string_array_utf8(self):
             df.to_csv(path, encoding='utf-8')
             with open(path, 'r') as f:
                 assert f.read() == expected_utf8
+
+    def test_to_csv_sparse_dataframe(self):
+        sdf = pd.SparseDataFrame({'a': [1, 2]})
+
+        with tm.ensure_clean('sparse_df.csv') as path:
+            sdf.to_csv(path, index=False)
+            df = pd.read_csv(path)
+
+            tm.assert_sp_frame_equal(df.to_sparse(), sdf)


Code review comments:
diff_hunk: @@ -704,7 +704,9 @@ def to_native_types(self, slicer=None, na_rep='nan', quoting=None,
                         **kwargs):
         """ convert to our native types format, slicing if desired """
 
-        values = self.values
+        # values = self.values
path: pandas/core/internals.py
text: @user1:
you can remove the comment

diff_hunk: @@ -253,3 +253,12 @@ def test_to_csv_string_array_utf8(self):
             df.to_csv(path, encoding='utf-8')
             with open(path, 'r') as f:
                 assert f.read() == expected_utf8
+
+    def test_to_csv_sparse_dataframe(self):
+        sdf = pd.SparseDataFrame({'a': [1, 2]})
path: pandas/tests/io/formats/test_to_csv.py
text: @user1:
add a comment here for the issue number. I would rather have this test in pandas/tests/sparse/frame/test_to_csv.py 

can you add several variations, e.g. using fill value and with some nulls.