ENH: Sparse int64 and bool dtype support enhancement
State: closed

- [x] closes #667, closes #8292, closes #13001, closes #8276, closes #13110
  - [x] tests added / passed
  - [x] passes `git diff upstream/master | flake8 --diff`
  - [x] whatsnew entry

Currently, sparse doesn't support `int64` and `bool` dtypes actually. When `int` or `bool` values are passed, it is coerced to `float64` if `dtype`kw  is not explicitly specified. 

**on current master**

```
pd.SparseArray([1, 2, 0, 0 ])
# [1.0, 2.0, 0.0, 0.0]
# Fill: nan
# IntIndex
# Indices: array([0, 1, 2, 3], dtype=int32)

pd.SparseArray([True, False, True])
# [1.0, 0.0, 1.0]
# Fill: nan
# IntIndex
# Indices: array([0, 1, 2], dtype=int32)
```

**after this PR**

The created data should have the `dtype` of passed values (as the same as normal `Series`). 

```
pd.SparseArray([1, 2, 0, 0 ])
# [1, 2, 0, 0]
# Fill: 0
# IntIndex
# Indices: array([0, 1], dtype=int32)

pd.SparseArray([True, False, True])
# [True, False, True]
# Fill: False
# IntIndex
# Indices: array([0, 2], dtype=int32)
```

Also, `fill_value` is automatically specified according to the following rules (because `np.nan` cannot appear in `int` or `bool` dtype):

Basic rule: sparse `dtype` must not be changed when it is converted to dense. 
- If `sparse_index` is specified and data has a hole (missing values):
  - `fill_value` is np.nan
  - `dtype` is `float64` or `object` (which can store both `data` and `fill_value`)
- If `sparse_index` is None (all values are provided via `data`, no missing values)
  - if `fill_value` is not explicitly passed, following default will be used depending on its dtype.
    - `float`: `np.nan`
    - `int`: `0`
    - `bool`: `False`


Labels:
name: Enhancement
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Sparse
description: Sparse Data Type

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13849?src=pr) is 85.27% (diff: 98.63%)

> Merging [#13849](https://codecov.io/gh/pydata/pandas/pull/13849?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #13849   diff @@
==========================================
  Files           139        139          
  Lines         50511      50523    +12   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43071      43083    +12   
  Misses         7440       7440          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [10bf721...341585a](https://codecov.io/gh/pydata/pandas/compare/10bf7213240eaa0960f50f86c7790d64d8973060...341585aa966ca9bd6dad98c819bc818e7d471e88?src=pr)

@sinhrks getting tons of warnings compiling on windows....all the same

```
pandas\src\sparse.c(63861) : warning C4244: '=' : conversion from 'Py_ssize_t' to '__pyx_t_5numpy_int32_t', possible loss of data
pandas\src\sparse.c(63870) : warning C4244: '=' : conversion from 'Py_ssize_t' to '__pyx_t_5numpy_int32_t', possible loss of data
pandas\src\sparse.c(66180) : warning C4244: '=' : conversion from 'Py_ssize_t' to '__pyx_t_5numpy_int32_t', possible loss of data
pandas\src\sparse.c(66189) : warning C4244: '=' : conversion from 'Py_ssize_t' to '__pyx_t_5numpy_int32_t', possible loss of data
pandas\src\sparse.c(68499) : warning C4244: '=' : conversion from 'Py_ssize_t' to '__pyx_t_5numpy_int32_t', possible loss of data
```

rebased and added the doc, now ready for review.

lgtm. just some doc corrections. ping on green.

@jreback Thx for review. One point is whether we should prohibit dtypes other than "relatively-well" supported ones (currently `float64`, after the PR `int64` and `bool` also). Maybe my title was misleading, thus change the title.

There are few issues which uses sparse data with `object` dtype  like #11633, #13917. So I feel we should keep the current behavior for other dtype rather than raise (may better to show a warning).

CC: @sstanovnik 

yeah `object` dtypes are only partially supported ATM. I think can leave that ok, though possibly we could warn, @jorisvandenbossche ?

Let me add my 2¢, since you went as far as CC-ing me. I can't make a very informed opinion, since I don't know enough about pandas' internals, and I obviously have an interest (biolab/orange3#1347) for supporting arbitrary types.

My thoughts are that you should be able to throw the same kind of data in a dense or a sparse DataFrame so that they are equivalent. An example off the top of my head is a SparseDataFrame with a recommendation dataset with rows as movies and columns as users, and additional metadata (string) columns about each movie. I don't know if this is possible, but judging from my time with BlockManager, you could maybe use dense string columns mixed in-between an otherwise sparse structure, if supporting sparse string storage is too hard. 

As I said, I may be completely off-target here, just some thoughts :)

Sorry, not familiar with sparse. But: using object dtype, does it work enough to use it for certain cases? If yes, I would not remove it.

@sinhrks Does this also close https://github.com/pydata/pandas/issues/13110?

I think object dtype can be used in some cases, but not fully sure as it is not tested well. Not remove ATM and add more tests to clarify (on another PR).

#13110 should be closed. Added whatsnew.

Disclaimer: I never used sparse or am familiar with the implementation (so my excuses if it is a stupid or naive question), but I quickly looked at the PR and have the following question.

Previously, for integer and boolean serieses, the 0 or False values were regarded as actual values, not an indication of 'not a value' in the sparse series. Isn't this a big change? (I don't know how much you could use it before this PR to be a problem) 
Next to that, having eg False for boolean arrays as the default fill_value also seems a bit strange to me. I would expect that somebody who wants a boolean sparse array, would want to be able to have both True and False values as actual values? (eg something like `[True, -, -, False, -, -, True]`)?
Of course this is currently because boolean serieses cannot have anything else as True or False.

OK, so probably my question should be categorized in the naive category :-) 
I see that this is the same as what `scipy.sparse` does, so seems like a sensible default then.

joris your example already works you can have any values u want as actual values (both True and False); the fill value is for the missing value indicator when I need to densify (it's the default)

so this is not a conceptual change at all just a change to keep dtype consistency 

@jreback I was looking at the `to_sparse` examples. So the fill_value is also used to convert from dense to sparse. So the output what you see there (eg in case of `pd.Series([1, 0, 0]).to_sparse()`) has changed (previously that was a block length of 3, now of 1). But no problem, I understand that the actual behaviour you want has not changed.

@jreback This PR for the rest OK to merge for you, Jeff? (it's closing a lot of issues for 0.19.0 :-))

@sinhrks Can you update the docstrings for SparseDataFrame, SparseSeries and SparseArray? They all still mention the fact that only floats are supported or that nan is the default fill value.

@sinhrks Thanks a lot!

@sinhrks appveyor started failing (some int dtype issues): 

```
======================================================================
FAIL: test_append_zero (pandas.sparse.tests.test_list.TestSparseList)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python27_64\envs\pandas\lib\site-packages\pandas\sparse\tests\test_list.py", line 64, in test_append_zero
    tm.assert_sp_array_equal(sparr, SparseArray(arr, fill_value=0))
  File "C:\Python27_64\envs\pandas\lib\site-packages\pandas\util\testing.py", line 1392, in assert_sp_array_equal
    assert_numpy_array_equal(left.sp_values, right.sp_values)
  File "C:\Python27_64\envs\pandas\lib\site-packages\pandas\util\testing.py", line 1083, in assert_numpy_array_equal
    assert_attr_equal('dtype', left, right, obj=obj)
  File "C:\Python27_64\envs\pandas\lib\site-packages\pandas\util\testing.py", line 878, in assert_attr_equal
    left_attr, right_attr)
  File "C:\Python27_64\envs\pandas\lib\site-packages\pandas\util\testing.py", line 1018, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: numpy array are different
Attribute "dtype" are different
[left]:  int64
[right]: int32
```

@jorisvandenbossche thx for pointing out, will fix.