Tests for Deprecate SparseArray for python 3.6 and 3.7 and fixes to other deprecation tests
State: closed

- [x] closes #30642
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry
   - N/A

Turns out that the existing tests were not testing deprecation correctly for both python 3.7 and python 3.6, so had to change some of the code in `test_api.py` to make that work right.

For `pd.SparseArray` in python 3.6, we can only issue a warning when the constructor `pd.SparseArray` is used.  But this is consistent with `pd.datetime` and `pd.np` with python 3.6, which will issue warnings when things like `pd.datetime.now()` are called.  However, for python 3.6, I could not figure out a way to issue a warning on `pd.datetime(2015, 10, 11, 0, 0)`, so we may just have to live with that.  




Labels:
name: Sparse
description: Sparse Data Type
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Some notes on the deprecation logic, and how it appears to users.  First, it appears that if you are running the python interpreter, you only get one deprecation warning per class, but if you use ipython, you get it each time.  

First, here are things with python.  Note that a repeat usage does not show the deprecation warning.  So I restart python each time to test different invocations.
```
(pandas-dev) c:\Code\pandas_dev\pandas> python
Python 3.7.6 | packaged by conda-forge | (default, Dec 26 2019, 23:30:11) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.__version__
'0.26.0.dev0+1686.g52a7d038c'
>>> pd.datetime.now()
__main__:1: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime module instead.
datetime.datetime(2020, 1, 7, 21, 47, 44, 406012)
>>> pd.datetime.now()
datetime.datetime(2020, 1, 7, 21, 47, 46, 291970)
>>> quit()

(pandas-dev) c:\Code\pandas_dev\pandas> python
Python 3.7.6 | packaged by conda-forge | (default, Dec 26 2019, 23:30:11) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.datetime(2015, 10, 12, 0, 0)
__main__:1: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime module instead.
datetime.datetime(2015, 10, 12, 0, 0)
>>> pd.datetime(2015, 10, 12, 0, 0)
datetime.datetime(2015, 10, 12, 0, 0)
>>> pd.np.array(10)
__main__:1: FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead
array(10)
>>> pd.np.array(10)
array(10)
>>> pd.SparseArray([1,2,3])
__main__:1: FutureWarning: The pandas.SparseArray class is deprecated and will be removed from pandas in a future version. Use pandas.arrays.SparseArray instead.
[1, 2, 3]
Fill: 0
IntIndex
Indices: array([0, 1, 2])

>>> pd.SparseArray([1,2,3])
[1, 2, 3]
Fill: 0
IntIndex
Indices: array([0, 1, 2])

>>> quit()
```
Now, with ipython, you get the deprecation warning each time.
```
(pandas-dev) c:\Code\pandas_dev\pandas> ipython
Python 3.7.6 | packaged by conda-forge | (default, Dec 26 2019, 23:30:11) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.10.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pandas as pd

In [2]: pd.datetime.now()
C:\Anaconda3\envs\pandas-dev\Scripts\ipython:1: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime module instead.
Out[2]: datetime.datetime(2020, 1, 7, 21, 54, 0, 906360)

In [3]: pd.datetime.now()
C:\Anaconda3\envs\pandas-dev\Scripts\ipython:1: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime module instead.
Out[3]: datetime.datetime(2020, 1, 7, 21, 54, 2, 491123)

In [4]: pd.datetime(2015, 10, 12, 0, 0)
C:\Anaconda3\envs\pandas-dev\Scripts\ipython:1: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime module instead.
Out[4]: datetime.datetime(2015, 10, 12, 0, 0)

In [5]: pd.np.array(12)
C:\Anaconda3\envs\pandas-dev\Scripts\ipython:1: FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead
Out[5]: array(12)

In [6]: pd.np.array(12)
C:\Anaconda3\envs\pandas-dev\Scripts\ipython:1: FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead
Out[6]: array(12)

In [7]: pd.SparseArray([1,2,3])
C:\Anaconda3\envs\pandas-dev\Scripts\ipython:1: FutureWarning: The pandas.SparseArray class is deprecated and will be removed from pandas in a future version. Use pandas.arrays.SparseArray instead.
Out[7]:
[1, 2, 3]
Fill: 0
IntIndex
Indices: array([0, 1, 2])

In [8]: pd.SparseArray([1,2,3])
C:\Anaconda3\envs\pandas-dev\Scripts\ipython:1: FutureWarning: The pandas.SparseArray class is deprecated and will be removed from pandas in a future version. Use pandas.arrays.SparseArray instead.
Out[8]:
[1, 2, 3]
Fill: 0
IntIndex
Indices: array([0, 1, 2])
```
Now, the results with python 3.6.  Note that I could not get `pd.datetime(2015, 10, 12, 0, 0)` to emit a warning.  It just fails.  This is because in python 3.6, we can't differentiate between using `datetime` as a constructor and accessing its class methods.
```

(pandas36) C:\Code\pandas_dev\pandas36> python
Python 3.6.7 (default, Dec  6 2019, 07:03:06) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.__version__
'0.26.0.dev0+1681.g2e0bab04c'
>>> pd.datetime.now()
__main__:1: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime instead.
datetime.datetime(2020, 1, 7, 22, 0, 3, 549109)
>>> pd.datetime.now()
datetime.datetime(2020, 1, 7, 22, 0, 5, 36133)
>>> quit()

(pandas36) C:\Code\pandas_dev\pandas36> python
Python 3.6.7 (default, Dec  6 2019, 07:03:06) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.datetime(2015, 10, 12, 0, 0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: '__Datetime' object is not callable
>>> pd.datetime.now()
__main__:1: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime instead.
datetime.datetime(2020, 1, 7, 22, 0, 36, 1357)
>>> pd.np.array(10)
__main__:1: FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead
array(10)
>>> pd.np.array(10)
array(10)
>>> pd.SparseArray([1,2,3])
__main__:1: FutureWarning: The pandas.SparseArray class is deprecated and will be removed from pandas in a future version. Use pandas.arrays.SparseArray instead.
[1, 2, 3]
Fill: 0
IntIndex
Indices: array([0, 1, 2])

>>> pd.SparseArray([1,2,3])
[1, 2, 3]
Fill: 0
IntIndex
Indices: array([0, 1, 2])

>>> quit()
```
And now ipython 3.6:
```

(pandas36) C:\Code\pandas_dev\pandas36> ipython
Python 3.6.7 (default, Dec  6 2019, 07:03:06) [MSC v.1900 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.11.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pandas as pd

In [2]: pd.datetime.now()
C:\Anaconda3\envs\pandas36\Scripts\ipython:1: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime instead.
Out[2]: datetime.datetime(2020, 1, 7, 22, 35, 20, 57781)

In [3]: pd.datetime.now()
C:\Anaconda3\envs\pandas36\Scripts\ipython:1: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime instead.
Out[3]: datetime.datetime(2020, 1, 7, 22, 35, 23, 415827)

In [4]: pd.datetime(2015, 10, 12, 0, 0)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-5ba670431981> in <module>
----> 1 pd.datetime(2015, 10, 12, 0, 0)

TypeError: '__Datetime' object is not callable

In [5]: pd.np.array(12)
C:\Anaconda3\envs\pandas36\Scripts\ipython:1: FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead
Out[5]: array(12)

In [6]: pd.np.array(12)
C:\Anaconda3\envs\pandas36\Scripts\ipython:1: FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead
Out[6]: array(12)

In [7]: pd.SparseArray([1,2,3])
C:\Anaconda3\envs\pandas36\Scripts\ipython:1: FutureWarning: The pandas.SparseArray class is deprecated and will be removed from pandas in a future version. Use pandas.arrays.SparseArray instead.
Out[7]:
[1, 2, 3]
Fill: 0
IntIndex
Indices: array([0, 1, 2])

In [8]: pd.SparseArray([1,2,3])
C:\Anaconda3\envs\pandas36\Scripts\ipython:1: FutureWarning: The pandas.SparseArray class is deprecated and will be removed from pandas in a future version. Use pandas.arrays.SparseArray instead.
Out[8]:
[1, 2, 3]
Fill: 0
IntIndex
Indices: array([0, 1, 2])

In [9]: quit()

```

So the behaviors (with this pull request) are what we want, and I just don't know how to handle the `pd.datetime(2015, 10, 12, 0, 0)` issue with python 3.6.

Also, given that `python` warns only once, and ipython warns multiple times, I don't know if the tests we created will catch multiple tests on the warnings.

My suggestion is that you merge in this pull request, and we could create an issue to track two things:
1. How to warn if someone uses `pd.datetime(2015, 10, 12, 0, 0)` when using python 3.6.
2. How to ensure that the tests in `pandas/tests/api/test_api.py` are properly testing the deprecations if it is the case that python will only warn once, i.e., make sure that multiple tests on warnings are actually caught.

A little bit more on the warnings if someone tries to import a deprecated class:
python 3.7 works fine because you can do `__getattr__()` at the module level:
```
Python 3.7.6 | packaged by conda-forge | (default, Dec 26 2019, 23:30:11) [MSC v
.1916 64 bit (AMD64)] on win32                                                  
Type "help", "copyright", "credits" or "license" for more information.          
>>> import pandas as pd                                                         
>>> from pandas import SparseArray as SparseArray                               
__main__:1: FutureWarning: The pandas.SparseArray class is deprecated and will b
e removed from pandas in a future version. Use pandas.arrays.SparseArray instead
.                                                                               
>>> from pandas import np as pnp                                                
__main__:1: FutureWarning: The pandas.np module is deprecated and will be remove
d from pandas in a future version. Import numpy directly instead                
>>> from pandas import datetime as ddt                                          
__main__:1: FutureWarning: The pandas.datetime class is deprecated and will be r
emoved from pandas in a future version. Import from datetime module instead.    
>>>                                                                             
```
python 3.6 doesn't warn when you import, but does warn when you use them:
```
Python 3.6.7 (default, Dec  6 2019, 07:03:06) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> from pandas import SparseArray as SparseArray
>>> from pandas import np as pnp
>>> from pandas import datetime as ddt
>>> SparseArray([1,2,3])
__main__:1: FutureWarning: The pandas.SparseArray class is deprecated and will be removed from pandas in a future version. Use pandas.arrays.SparseArray instead.
[1, 2, 3]
Fill: 0
IntIndex
Indices: array([0, 1, 2])

>>> pnp.array(10)
__main__:1: FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead
array(10)
>>> ddt.now()
__main__:1: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime instead.
datetime.datetime(2020, 1, 7, 22, 47, 38, 129842)
>>>
```


So IIUC, the only deprecation that's working correctly on master for py 3.6 is `np`.

```python
>>> pd.datetime
<class 'datetime.datetime'>
>>> pd.datetime.now()
datetime.datetime(2020, 1, 8, 9, 29, 32, 925293)
>>> pd.np
<pandas.__numpy object at 0x109f016d8>
>>> pd.np.ones(4)
__main__:1: FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead
array([1., 1., 1., 1.])
```

essentially, the deprecation strategy we have for 3.6 doesn't work well for classes like SparseArray and `datetime.datetime`.
@TomAugspurger @jorisvandenbossche I figured out how to make `datetime` work with python 3.6 that preserves functionality and issues warnings when using `pd.datetime(2015, 11, 2, 0, 0)` and `pd.datetime.now()`.  It's in my latest commit 5c59f84 .  Had to figure out how metaclasses work.

Here is what it looks like in ipython:
```
Python 3.6.7 (default, Dec  6 2019, 07:03:06) [MSC v.1900 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.11.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pandas as pd

In [2]: pd.__version__
Out[2]: '0.26.0.dev0+1682.g5c59f841f'

In [3]: pd.datetime(2015, 11, 12, 0, 0)
C:\Anaconda3\envs\pandas36\Scripts\ipython:1: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime instead.
Out[3]: datetime.datetime(2015, 11, 12, 0, 0)

In [4]: pd.datetime.now()
C:\Anaconda3\envs\pandas36\Scripts\ipython:1: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime instead.
Out[4]: datetime.datetime(2020, 1, 8, 12, 39, 54, 675891)
```
https://stackoverflow.com/questions/13135712/class-method-instancecheck-does-not-work

basically implement ``__instancecheck__`` in the metaclass
@jorisvandenbossche @TomAugspurger I think I have it right now for datetime and SparseArray.  This is in commit f1b89f4. Here's what things look like in python 3.6 using ipython:
```python
Python 3.6.7 (default, Dec  6 2019, 07:03:06) [MSC v.1900 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.11.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pandas as pd

In [2]: pd.datetime(2018, 10, 11, 2, 3)
C:\Anaconda3\envs\pandas36\Scripts\ipython:1: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime instead.
Out[2]: datetime.datetime(2018, 10, 11, 2, 3)

In [3]: pd.datetime.now()
C:\Anaconda3\envs\pandas36\Scripts\ipython:1: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime instead.
Out[3]: datetime.datetime(2020, 1, 8, 16, 16, 4, 650619)

In [4]: isinstance(pd.datetime(2018, 10, 11, 2, 3), pd.datetime)
C:\Anaconda3\envs\pandas36\Scripts\ipython:1: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime instead.
Out[4]: True

In [5]: pd.SparseArray([1,2,3])
C:\Anaconda3\envs\pandas36\Scripts\ipython:1: FutureWarning: The pandas.SparseArray class is deprecated and will be removed from pandas in a future version. Use pandas.arrays.SparseArray instead.
Out[5]:
[1, 2, 3]
Fill: 0
IntIndex
Indices: array([0, 1, 2])

In [6]: isinstance(pd.array([1, 2, 3], dtype="Sparse"), pd.SparseArray)
Out[6]: True
```
And here it is for python 3.7:
```python
Python 3.7.6 | packaged by conda-forge | (default, Dec 26 2019, 23:30:11) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.10.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pandas as pd

In [2]: pd.datetime(2018, 10, 11, 2, 3)
C:\Anaconda3\envs\pandas-dev\Scripts\ipython:1: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime module instead.
Out[2]: datetime.datetime(2018, 10, 11, 2, 3)

In [3]: pd.datetime.now()
C:\Anaconda3\envs\pandas-dev\Scripts\ipython:1: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime module instead.
Out[3]: datetime.datetime(2020, 1, 8, 16, 19, 39, 764998)

In [4]: isinstance(pd.datetime(2018, 10, 11, 2, 3), pd.datetime)
C:\Anaconda3\envs\pandas-dev\Scripts\ipython:1: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime module instead.
Out[4]: True

In [5]: pd.SparseArray([1,2,3])
C:\Anaconda3\envs\pandas-dev\Scripts\ipython:1: FutureWarning: The pandas.SparseArray class is deprecated and will be removed from pandas in a future version. Use pandas.arrays.SparseArray instead.
Out[5]:
[1, 2, 3]
Fill: 0
IntIndex
Indices: array([0, 1, 2])

In [6]: isinstance(pd.array([1, 2, 3], dtype="Sparse"), pd.SparseArray)
C:\Anaconda3\envs\pandas-dev\Scripts\ipython:1: FutureWarning: The pandas.SparseArray class is deprecated and will be removed from pandas in a future version. Use pandas.arrays.SparseArray instead.
Out[6]: True
```

The one difference is now in python 3.6, if you refer to `pd.SparseArray` or `pd.datetime`, you will not get a warning, whereas you do get it in python 3.7.

@jreback all green
lgtm. ping on green.
@Dr-Irv looks like you *do* need those assignments to make mypy happy.
> @Dr-Irv looks like you _do_ need those assignments to make mypy happy.

@jreback I knew there was something.  Reverted it back.
thanks @Dr-Irv ok ping on green.
@jreback all green
thanks @Dr-Irv