Repository: pandas-dev/pandas

Issue:
BUG: SparseDataFrame from scipy.sparse.dok_matrix returns incorrect dataframe in python 2.7
#### Code Sample, a copy-pastable example if possible
Python 2.7
```python
>>> import sys
>>> sys.version
'2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:05:08) \n[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]'
>>> import numpy as np
>>> import scipy.sparse
>>> a = np.arange(1, 5).reshape(2,2)
>>> a
array([[1, 2],
       [3, 4]])
>>> spm = scipy.sparse.dok_matrix(a)
>>> spm
<2x2 sparse matrix of type '<type 'numpy.int64'>'
	with 4 stored elements in Dictionary Of Keys format>
>>> pd.SparseDataFrame(a)
   0  1
0  1  2
1  3  4
>>> pd.SparseDataFrame(spm)
   0  1
0  3  2
1  1  4
```
Python 3.6
```python
>>> import sys
>>> sys.version
'3.6.1 |Continuum Analytics, Inc.| (default, Mar 22 2017, 19:25:17) \n[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]'
>>> a = np.arange(1, 5).reshape(2,2)
>>> spm = scipy.sparse.dok_matrix(a)
>>> pd.SparseDataFrame(spm)
   0  1
0  1  2
1  3  4
```

#### Problem description

Initialization of SparseDataFrame with scipy.sparse.dok_matrix returns a different result from direct initialization from np.ndarray only in Python 2.7. This is inconsistent and unestimated behavior.
The scipy.sparse.dia_matrix shows similar buggy behavior.

#### Expected Output
```python
>>> spm = scipy.sparse.dok_matrix(np.arange(1, 5).reshape(2,2))
>>> pd.SparseDataFrame(spm)
   0  1
0  1  2
1  3  4
```

#### Output of ``pd.show_versions()``

<details>
INSTALLED VERSIONS
------------------
commit: 075eca1fa085359b71f26d02c2f44156068e123c
python: 2.7.13.final.0
python-bits: 64
OS: Darwin
OS-release: 16.5.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: ja_JP.UTF-8
LOCALE: None.None

pandas: 0.20.0rc1+29.g075eca1
pytest: 3.0.7
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.18.1
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
pandas_gbq: None
pandas_datareader: None
</details>



Discussion:
Do you have the same pandas and scipy versions on both?
Yes, I have scipy version 0.18.1 installed on both environments (Python 3.6 and Python 2.7)
cc @kernc 

Resolving patch:
diff --git a/pandas/core/sparse/frame.py b/pandas/core/sparse/frame.py
--- a/pandas/core/sparse/frame.py
+++ b/pandas/core/sparse/frame.py
@@ -190,8 +190,8 @@ def _init_spmatrix(self, data, index, columns, dtype=None,
         values = Series(data.data, index=data.row, copy=False)
         for col, rowvals in values.groupby(data.col):
             # get_blocks expects int32 row indices in sorted order
+            rowvals = rowvals.sort_index()
             rows = rowvals.index.values.astype(np.int32)
-            rows.sort()
             blocs, blens = get_blocks(rows)
 
             sdict[columns[col]] = SparseSeries(


Test patch:
diff --git a/pandas/tests/sparse/test_frame.py b/pandas/tests/sparse/test_frame.py
--- a/pandas/tests/sparse/test_frame.py
+++ b/pandas/tests/sparse/test_frame.py
@@ -1146,8 +1146,8 @@ def test_isnotnull(self):
         tm.assert_frame_equal(res.to_dense(), exp)
 
 
-@pytest.mark.parametrize('index', [None, list('ab')])  # noqa: F811
-@pytest.mark.parametrize('columns', [None, list('cd')])
+@pytest.mark.parametrize('index', [None, list('abc')])  # noqa: F811
+@pytest.mark.parametrize('columns', [None, list('def')])
 @pytest.mark.parametrize('fill_value', [None, 0, np.nan])
 @pytest.mark.parametrize('dtype', [bool, int, float, np.uint16])
 def test_from_to_scipy(spmatrix, index, columns, fill_value, dtype):
@@ -1156,7 +1156,9 @@ def test_from_to_scipy(spmatrix, index, columns, fill_value, dtype):
 
     # Make one ndarray and from it one sparse matrix, both to be used for
     # constructing frames and comparing results
-    arr = np.eye(2, dtype=dtype)
+    arr = np.eye(3, dtype=dtype)
+    # GH 16179
+    arr[0, 1] = dtype(2)
     try:
         spm = spmatrix(arr)
         assert spm.dtype == arr.dtype
@@ -1245,6 +1247,26 @@ def test_from_to_scipy_object(spmatrix, fill_value):
     assert sdf.to_coo().dtype == res_dtype
 
 
+def test_from_scipy_correct_ordering(spmatrix):
+    # GH 16179
+    tm.skip_if_no_package('scipy')
+
+    arr = np.arange(1, 5).reshape(2, 2)
+    try:
+        spm = spmatrix(arr)
+        assert spm.dtype == arr.dtype
+    except (TypeError, AssertionError):
+        # If conversion to sparse fails for this spmatrix type and arr.dtype,
+        # then the combination is not currently supported in NumPy, so we
+        # can just skip testing it thoroughly
+        return
+
+    sdf = pd.SparseDataFrame(spm)
+    expected = pd.SparseDataFrame(arr)
+    tm.assert_sp_frame_equal(sdf, expected)
+    tm.assert_frame_equal(sdf.to_dense(), expected.to_dense())
+
+
 class TestSparseDataFrameArithmetic(object):
 
     def test_numeric_op_scalar(self):