Repository: pandas-dev/pandas
Pull request: TST Add test for loc on sparse dataframes
Adds a test for loc on sparse dataframes.

- [x] closes #34687
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


Problem statement:
BUG: multiple application of .loc on sparse DataFrame results in NaN and filling the DataFrame 
- [x] I have checked that this issue has not already been reported.

- [x] I have confirmed this bug exists on the latest version of pandas.

- [x] (optional) I have confirmed this bug exists on the master branch of pandas.

---

#### Code Sample, a copy-pastable example

```python
# Your code here

from scipy.sparse import coo_matrix, eye  
import pandas as pd

df = pd.DataFrame.sparse.from_spmatrix(eye(10))

df.sparse.density 
# Should output 0.1

df.loc[range(5)]                                                        
#     0    1    2    3    4    5    6    7    8    9
# 0  1.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
# 1  0.0  1.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
# 2  0.0  0.0  1.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
# 3  0.0  0.0  0.0  1.0  0.0  0.0  0.0  0.0  0.0  0.0
# 4  0.0  0.0  0.0  0.0  1.0  0.0  0.0  0.0  0.0  0.0

df.loc[range(5)].sparse.density
# outputs 0.1

df.loc[range(5)].loc[range(3)]                                          
#     0    1    2    3    4   5   6   7   8   9
# 0  1.0  0.0  0.0  0.0  0.0 NaN NaN NaN NaN NaN
# 1  0.0  1.0  0.0  0.0  0.0 NaN NaN NaN NaN NaN
# 2  0.0  0.0  1.0  0.0  0.0 NaN NaN NaN NaN NaN

df.loc[range(5)].loc[range(3)].sparse.density
# outputs 0.6

```

#### Problem description

It seems that sparse DataFrame extracted using loc does not behave as the original ones. This creates inconsistencies in our processing pipelines, depending on filtering and selection that has been applied, sometimes producing "Nans", severely impacting memory consumption and computational time. 

#### Expected Output

The output of loc should not depend on multiple slicing, i.e.

df.loc[range(5)].loc[range(3)] = df.loc[range(3)]

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None

pandas           : 1.0.4
numpy            : 1.18.5
pytz             : 2020.1
dateutil         : 2.8.1
pip              : 9.0.3
setuptools       : 39.0.1
Cython           : 0.29.19
pytest           : 5.4.3
hypothesis       : 5.16.1
sphinx           : 3.1.0
blosc            : 1.9.1
feather          : None
xlsxwriter       : 1.2.9
lxml.etree       : 4.5.1
html5lib         : 1.0.1
pymysql          : None
psycopg2         : None
jinja2           : 2.11.2
IPython          : 7.15.0
pandas_datareader: None
bs4              : 4.9.1
bottleneck       : 1.3.2
fastparquet      : 0.4.0
gcsfs            : None
lxml.etree       : 4.5.1
matplotlib       : 3.2.1
numexpr          : 2.7.1
odfpy            : None
openpyxl         : 3.0.3
pandas_gbq       : None
pyarrow          : 0.17.1
pytables         : None
pytest           : 5.4.3
pyxlsb           : None
s3fs             : 0.4.2
scipy            : 1.4.1
sqlalchemy       : 1.3.17
tables           : 3.6.1
tabulate         : 0.8.7
xarray           : 0.15.1
xlrd             : 1.2.0
xlwt             : 1.3.0
xlsxwriter       : 1.2.9
numba            : 0.49.1

</details>

Hints from issue discussion:
On master I have

```python
In [41]: df.loc[range(5)].loc[range(3)].sparse.density
Out[41]: 0.1
```

But for some reason the last columns are cast to Sparse[int] dtype

```python
In [53]: df.loc[range(5)].loc[range(3)].dtypes
Out[53]:
0    Sparse[float64, 0]
1    Sparse[float64, 0]
2    Sparse[float64, 0]
3    Sparse[float64, 0]
4    Sparse[float64, 0]
5      Sparse[int64, 0]
6      Sparse[int64, 0]
7      Sparse[int64, 0]
8      Sparse[int64, 0]
9      Sparse[int64, 0]
dtype: object
```

Not sure what's going on.
Quite strange that we have different results and you cannot reproduce my outputs. 

Here is the commit I have installed (bbb89cad61ba030b1b6790bb052f64241492060d). 

```
Python 3.6.5 (default, Jan 14 2020, 14:04:16) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.15.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pandas as pd                                                                                                          

In [2]: from scipy.sparse import coo_matrix, eye                                                                                     

In [3]: df = pd.DataFrame.sparse.from_spmatrix(eye(10))                                                                              

In [4]: df.loc[range(5)].loc[range(3)]                                                                                               
Out[4]: 
     0    1    2    3    4   5   6   7   8   9
0  1.0  0.0  0.0  0.0  0.0 NaN NaN NaN NaN NaN
1  0.0  1.0  0.0  0.0  0.0 NaN NaN NaN NaN NaN
2  0.0  0.0  1.0  0.0  0.0 NaN NaN NaN NaN NaN

In [5]: exit         
```

Not sure it would matter but I'm using python 3.6.5 on a mac osx 10.15.2. 

BTW, the issue about the data-type you got might be related to this other issue #34540
I was looking into this issue and must add it happens when the second `.loc` lookup is an iterable, but it's fine with slices. I mean:
```
In [1]: df.loc[[0, 1, 2, 3, 4, 5]].loc[[3,4,5]]                                                                                           
Out[1]: 
     0    1    2    3    4    5   6   7   8   9
3  0.0  0.0  0.0  1.0  0.0  0.0 NaN NaN NaN NaN
4  0.0  0.0  0.0  0.0  1.0  0.0 NaN NaN NaN NaN
5  0.0  0.0  0.0  0.0  0.0  1.0 NaN NaN NaN NaN

In [2]: df.loc[[0, 1, 2, 3, 4, 5]].loc[:3]  # or df.loc[range(6)].loc[:3]
Out[2]: 
     0    1    2    3    4    5    6    7    8    9
0  1.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
1  0.0  1.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
2  0.0  0.0  1.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
3  0.0  0.0  0.0  1.0  0.0  0.0  0.0  0.0  0.0  0.0
```

Another observation is that the number of columns it keeps depends in the first iloc lookup. See:

```
In [1]: df.loc[range(3)].loc[range(3)]                                                                                                   
Out[1]: 
     0    1    2   3   4   5   6   7   8   9
0  1.0  0.0  0.0 NaN NaN NaN NaN NaN NaN NaN
1  0.0  1.0  0.0 NaN NaN NaN NaN NaN NaN NaN
2  0.0  0.0  1.0 NaN NaN NaN NaN NaN NaN NaN
In [2]: df.loc[range(4)].loc[range(3)]                                                                                                   
Out[2]: 
     0    1    2    3   4   5   6   7   8   9
0  1.0  0.0  0.0  0.0 NaN NaN NaN NaN NaN NaN
1  0.0  1.0  0.0  0.0 NaN NaN NaN NaN NaN NaN
2  0.0  0.0  1.0  0.0 NaN NaN NaN NaN NaN NaN
In [3]: df.loc[range(8)].loc[range(3)]                                                                                                   
Out[3]: 
     0    1    2    3    4    5    6    7   8   9
0  1.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 NaN NaN
1  0.0  1.0  0.0  0.0  0.0  0.0  0.0  0.0 NaN NaN
2  0.0  0.0  1.0  0.0  0.0  0.0  0.0  0.0 NaN NaN
```
This works now on master. Dtypes are fine and results equal expected. May need tests
@TomAugspurger @MJafarMashhadi @deusebio @phofl @jd 
Hi Guy Have created test cases

Commit message: TST Add test for loc on sparse dataframes

Patch to review:
diff --git a/pandas/tests/indexing/test_loc.py b/pandas/tests/indexing/test_loc.py
index 85accac5a8235..a59319d31fec2 100644
--- a/pandas/tests/indexing/test_loc.py
+++ b/pandas/tests/indexing/test_loc.py
@@ -1163,6 +1163,25 @@ def test_loc_getitem_listlike_all_retains_sparse(self):
         result = df.loc[[0, 1]]
         tm.assert_frame_equal(result, df)
 
+    @td.skip_if_no_scipy
+    def test_loc_sparse_frame(self):
+        # GH34687
+        from scipy.sparse import eye
+
+        df = DataFrame.sparse.from_spmatrix(eye(5))
+        result = df.loc[range(2)]
+        expected = DataFrame(
+            [[1.0, 0.0, 0.0, 0.0, 0.0], [0.0, 1.0, 0.0, 0.0, 0.0]],
+            dtype=SparseDtype("float64", 0.0),
+        )
+        tm.assert_frame_equal(result, expected)
+
+        result = df.loc[range(2)].loc[range(1)]
+        expected = DataFrame(
+            [[1.0, 0.0, 0.0, 0.0, 0.0]], dtype=SparseDtype("float64", 0.0)
+        )
+        tm.assert_frame_equal(result, expected)
+
     @pytest.mark.parametrize("key_type", [iter, np.array, Series, Index])
     def test_loc_getitem_iterable(self, float_frame, key_type):
         idx = key_type(["A", "B", "C"])


Code review comments:
diff_hunk: @@ -1163,6 +1163,25 @@ def test_loc_getitem_listlike_all_retains_sparse(self):
         result = df.loc[[0, 1]]
         tm.assert_frame_equal(result, df)
 
+    @td.skip_if_no_scipy
+    def test_loc_sparse_frame(self):
path: pandas/tests/indexing/test_loc.py
text: @user1:
can you start the name with `test_loc_getitem` to match the pattern in this clas

mainly this is testing that .loc indexing retains SparseDtype right?  should test for Series too?

@author:
Have updated the name, and also added similar tests for sparse series, or would it be better to brake these out into a seperate function?

@user2:
yeah i think if you can move the series tests to series/indexing/test_indexing

@author:
I have moved them to that folder.

@user1:
sorry fellas, rule of thumb: if you're testing .loc, it belongs in tests.indexing.test_loc.  See https://github.com/pandas-dev/pandas/blob/master/doc/source/development/test_writing.rst

@author:
No problem, have now moved it back to `tests.indexing.test_loc`.