CLN GH23123 Move SparseArray to arrays
State: closed

- [X] closes #23123 
- [X] tests added / passed
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Importing `SparseArray` from the `Arrays` folder was causing circular import paths with the `__init__.py`, so I had to empty it and fix every other file that imported from `pandas.core.arrays`

Labels:
name: Refactor
description: Internal refactoring of code
name: Sparse
description: Sparse Data Type

Comments:
This change in imports of other arrays everywhere should not be needed. Can you detail where the circular import problem was coming from?
@JustinZhengBC you've got some `print()` calls in your PR that I guess were for your own debugging. Can you remove them?
The main import loop problem is in `pandas.core.types.dtypes.common.py`, which imports `SparseDtype` from `pandas.core.arrays.sparse.dtype` for two uses in checking `is_instance`. Whenever a SparseArray is imported, it now executes `pandas.core.arrays.__init__.py`. Many of the files mentioned in `__init__.py` in turn import `common.py`, causing another import from SparseArray, causing another call to `__init__.py` in the process.

I will try putting the offending import statements from `common.py` into the functions where they are used so that they are not automatically called on import, and see if that gets rid of the loop.
Almost all of the tests seem to be failing in the same place. The trace is below, and the offending line is `from pandas.core.arrays.sparse.array import _maybe_to_sparse`, where the tests claim they can't find `pandas.core.arrays.sparse.array`. The weird thing is, the tests do not fail locally, and I can even execute the exact same line without any errors, so I'm a bit stuck here. Am I doing something wrong, or is it that moving a folder and thus changing the API causes tests to fail because of how they're set up?

```
Traceback (most recent call last):
  File "/home/travis/miniconda3/envs/pandas/lib/python2.7/site-packages/_pytest/config/__init__.py", line 419, in _importconftest
    mod = conftestpath.pyimport()
  File "/home/travis/miniconda3/envs/pandas/lib/python2.7/site-packages/py/_path/local.py", line 668, in pyimport
    __import__(modname)
  File "/home/travis/build/pandas-dev/pandas/pandas/__init__.py", line 42, in <module>
    from pandas.core.api import *
  File "/home/travis/build/pandas-dev/pandas/pandas/core/api.py", line 10, in <module>
    from pandas.core.groupby import Grouper
  File "/home/travis/build/pandas-dev/pandas/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.groupby import GroupBy  # flake8: noqa
  File "/home/travis/build/pandas-dev/pandas/pandas/core/groupby/groupby.py", line 39, in <module>
    from pandas.core.generic import NDFrame
  File "/home/travis/build/pandas-dev/pandas/pandas/core/generic.py", line 43, in <module>
    from pandas.core.internals import BlockManager
  File "/home/travis/build/pandas-dev/pandas/pandas/core/internals/__init__.py", line 10, in <module>
    from .managers import (  # noqa:F401
  File "/home/travis/build/pandas-dev/pandas/pandas/core/internals/managers.py", line 32, in <module>
    from pandas.core.arrays.sparse.array import _maybe_to_sparse
ImportError: No module named sparse.array
ERROR: could not load /home/travis/build/pandas-dev/pandas/pandas/conftest.py
```
Did you try to simply `import pandas`? I think there is a wrong import in an `__init__.py`, or something similar.
Yes, `import pandas` works locally. I can even run the exact same lines in the .sh file that Travis fails on.

EDIT: I've noticed all the tests that run on python 2.7 fail said line, but the tests that run on 3.x don't. I'll look into this further.

EDIT2: Seems like somehow the `__init__.py` got lost in the move. Hopefully the tests work now.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23147?src=pr&el=h1) Report
> Merging [#23147](https://codecov.io/gh/pandas-dev/pandas/pull/23147?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/85dc1713bc6a1064f4afdf2a907bc9c72cdc364b?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `92.68%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/23147/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/23147?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #23147      +/-   ##
==========================================
+ Coverage   92.13%   92.14%   +0.01%     
==========================================
  Files         170      170              
  Lines       51073    51017      -56     
==========================================
- Hits        47056    47011      -45     
+ Misses       4017     4006      -11
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.57% <92.68%> (+0.01%)` | :arrow_up: |
| #single | `42.3% <68.29%> (+0.02%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/23147?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/arrays/period.py](https://codecov.io/gh/pandas-dev/pandas/pull/23147/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3BlcmlvZC5weQ==) | `95.56% <ø> (+1.27%)` | :arrow_up: |
| [pandas/compat/pickle\_compat.py](https://codecov.io/gh/pandas-dev/pandas/pull/23147/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9waWNrbGVfY29tcGF0LnB5) | `75.6% <ø> (ø)` | :arrow_up: |
| [pandas/core/util/hashing.py](https://codecov.io/gh/pandas-dev/pandas/pull/23147/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdXRpbC9oYXNoaW5nLnB5) | `98.4% <ø> (ø)` | :arrow_up: |
| [pandas/core/arrays/sparse/scipy\_sparse.py](https://codecov.io/gh/pandas-dev/pandas/pull/23147/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3NwYXJzZS9zY2lweV9zcGFyc2UucHk=) | `97.05% <ø> (ø)` | |
| [pandas/core/arrays/sparse/dtype.py](https://codecov.io/gh/pandas-dev/pandas/pull/23147/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3NwYXJzZS9kdHlwZS5weQ==) | `100% <ø> (ø)` | |
| [pandas/core/arrays/sparse/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/23147/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3NwYXJzZS9mcmFtZS5weQ==) | `94.86% <100%> (ø)` | |
| [pandas/io/packers.py](https://codecov.io/gh/pandas-dev/pandas/pull/23147/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhY2tlcnMucHk=) | `88.04% <100%> (ø)` | :arrow_up: |
| [pandas/core/arrays/integer.py](https://codecov.io/gh/pandas-dev/pandas/pull/23147/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2ludGVnZXIucHk=) | `94.9% <100%> (ø)` | :arrow_up: |
| [pandas/core/reshape/reshape.py](https://codecov.io/gh/pandas-dev/pandas/pull/23147/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9yZXNoYXBlLnB5) | `99.55% <100%> (ø)` | :arrow_up: |
| [pandas/core/arrays/sparse/array.py](https://codecov.io/gh/pandas-dev/pandas/pull/23147/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3NwYXJzZS9hcnJheS5weQ==) | `91.55% <100%> (ø)` | |
| ... and [24 more](https://codecov.io/gh/pandas-dev/pandas/pull/23147/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23147?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23147?src=pr&el=footer). Last update [85dc171...120bc5e](https://codecov.io/gh/pandas-dev/pandas/pull/23147?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@JustinZhengBC It seems you have removed the sparse tests, without adding them back in the `tests/arrays` folder.

Something general, should we keep the SparseSeries / SparseDataFrame where they were in `pandas.core.sparse` ? Since they are not really arrays, and we might deprecate them anyhow. Then moving them is maybe not that worth it?
@jorisvandenbossche thanks for the catch. All the tests should be there now. The "files changed" tab still says I deleted `pandas/core/sparse/api.py` even though it also says I created another file with the same name in the right location. Other than that, it shouldn't say I deleted any other files now.
Test failure is unrelated to the changes here. Apparently, the SparseArray.unique implementation is flaky. Taking a look now.
```python

In [4]: pd.SparseArray([0, 0]).unique()
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-4-c1befb06646c> in <module>
----> 1 pd.SparseArray([0, 0]).unique()

~/sandbox/pandas/pandas/core/sparse/array.py in unique(self)
    570     def unique(self):
    571         uniques = list(pd.unique(self.sp_values))
--> 572         fill_loc = self._first_fill_value_loc()
    573         if fill_loc >= 0:
    574             uniques.insert(fill_loc, self.fill_value)

~/sandbox/pandas/pandas/core/sparse/array.py in _first_fill_value_loc(self)
    562
    563         indices = self.sp_index.to_int_index().indices
--> 564         if indices[0] > 0:
    565             return 0
    566

IndexError: index 0 is out of bounds for axis 0 with size 0

```
And a fix

```diff
diff --git a/pandas/core/sparse/array.py b/pandas/core/sparse/array.py
index 15b5118db..17e2bd188 100644
--- a/pandas/core/sparse/array.py
+++ b/pandas/core/sparse/array.py
@@ -561,7 +561,7 @@ class SparseArray(PandasObject, ExtensionArray, ExtensionOpsMixin):
             return -1
 
         indices = self.sp_index.to_int_index().indices
-        if indices[0] > 0:
+        if len(indices) == 0 or indices[0] > 0:
             return 0
 
         diff = indices[1:] - indices[:-1]
diff --git a/pandas/tests/sparse/test_array.py b/pandas/tests/sparse/test_array.py
index 0257d9962..5b1afdc7a 100644
--- a/pandas/tests/sparse/test_array.py
+++ b/pandas/tests/sparse/test_array.py
@@ -1065,6 +1065,13 @@ def test_unique_na_fill(arr, fill_value):
     tm.assert_numpy_array_equal(a, b)
 
 
+def test_unique_all_sparse():
+    arr = SparseArray([0, 0])
+    result = arr.unique()
+    expected = SparseArray([0])
+    tm.assert_sp_array_equal(result, expected)
+
+
 def test_map():
     arr = SparseArray([0, 1, 2])
     expected = SparseArray([10, 11, 12], fill_value=10)
```

@jreback @jorisvandenbossche any objections to including that fix here? It seems simple enough to not need another PR, which will create a merge conflict here. Or we can merge this and do that in a separate PR.
Found another random failure. Going to split these two out to a new PR.
@JustinZhengBC I'm going to push some changes to your branch.
Hello @JustinZhengBC! Thanks for updating the PR.

 - There are no PEP8 issues in the file [`pandas/api/extensions/__init__.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/api/extensions/__init__.py) !

 - There are no PEP8 issues in the file [`pandas/compat/numpy/function.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/compat/numpy/function.py) !

 - There are no PEP8 issues in the file [`pandas/compat/pickle_compat.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/compat/pickle_compat.py) !

 - There are no PEP8 issues in the file [`pandas/core/arrays/__init__.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/core/arrays/__init__.py) !

 - There are no PEP8 issues in the file [`pandas/core/arrays/datetimes.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/core/arrays/datetimes.py) !

 - There are no PEP8 issues in the file [`pandas/core/arrays/integer.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/core/arrays/integer.py) !

 - There are no PEP8 issues in the file [`pandas/core/arrays/interval.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/core/arrays/interval.py) !

 - There are no PEP8 issues in the file [`pandas/core/arrays/period.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/core/arrays/period.py) !

 - There are no PEP8 issues in the file [`pandas/core/arrays/sparse.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/core/arrays/sparse.py) !

 - There are no PEP8 issues in the file [`pandas/core/dtypes/common.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/core/dtypes/common.py) !

 - There are no PEP8 issues in the file [`pandas/core/dtypes/concat.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/core/dtypes/concat.py) !

 - There are no PEP8 issues in the file [`pandas/core/frame.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/core/frame.py) !

 - There are no PEP8 issues in the file [`pandas/core/groupby/generic.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/core/groupby/generic.py) !

 - There are no PEP8 issues in the file [`pandas/core/internals/managers.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/core/internals/managers.py) !

 - There are no PEP8 issues in the file [`pandas/core/ops.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/core/ops.py) !

 - There are no PEP8 issues in the file [`pandas/core/reshape/reshape.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/core/reshape/reshape.py) !

 - There are no PEP8 issues in the file [`pandas/core/series.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/core/series.py) !

 - There are no PEP8 issues in the file [`pandas/core/sparse/api.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/core/sparse/api.py) !

 - There are no PEP8 issues in the file [`pandas/core/sparse/frame.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/core/sparse/frame.py) !

 - There are no PEP8 issues in the file [`pandas/core/sparse/series.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/core/sparse/series.py) !

 - There are no PEP8 issues in the file [`pandas/core/util/hashing.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/core/util/hashing.py) !

 - There are no PEP8 issues in the file [`pandas/io/packers.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/io/packers.py) !

 - There are no PEP8 issues in the file [`pandas/io/pytables.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/io/pytables.py) !

 - There are no PEP8 issues in the file [`pandas/tests/arrays/sparse/test_array.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/tests/arrays/sparse/test_array.py) !

 - There are no PEP8 issues in the file [`pandas/tests/arrays/sparse/test_libsparse.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/tests/arrays/sparse/test_libsparse.py) !

 - There are no PEP8 issues in the file [`pandas/tests/arrays/test_datetimelike.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/tests/arrays/test_datetimelike.py) !

 - There are no PEP8 issues in the file [`pandas/tests/extension/test_sparse.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/tests/extension/test_sparse.py) !

 - There are no PEP8 issues in the file [`pandas/tests/indexing/test_indexing.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/tests/indexing/test_indexing.py) !

 - There are no PEP8 issues in the file [`pandas/tests/series/test_combine_concat.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/tests/series/test_combine_concat.py) !

 - There are no PEP8 issues in the file [`pandas/tests/series/test_subclass.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/tests/series/test_subclass.py) !

 - There are no PEP8 issues in the file [`pandas/tests/sparse/series/test_series.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/tests/sparse/series/test_series.py) !

 - There are no PEP8 issues in the file [`pandas/tests/test_base.py`](https://github.com/JustinZhengBC/pandas/blob/1d0b50b295a472191c74fe341bb4b6fdd5cf6ffb/pandas/tests/test_base.py) !


Updates:

1. Moved pandas/core/arrays/sparse/array.py to pandas/core/arrays/sparse.py
2. Moved SparseDtype implementation into `pandas/core/arrays/sparse.py`
3. Moved `sparse/{series,frame,api,scipy_sparse}.py ` back to `pandas/core/sparse` (so no change from master)
I reverted some extraneous changes in fe53b50. @JustinZhengBC if you're interested, you can re-revert that commit later and make a separate PR with those sytlisitc changes. I notice now that it has a formatting change too in https://github.com/pandas-dev/pandas/pull/23147/files#diff-6cdf34ac065f3e1e350f94c70e76898bL6, sorry. You'll need to manually fix that if you re-revert it.
@jreback @jorisvandenbossche @datapythonista could I get a quick sanity check on this? I'd like to merge on green since I'll have some followup PRs that will be blocked by this.
lgtm at a glance
Looks good to me as well based on a quick view
OK to ignore the numpydev failure, or will we be re-running PRs after https://github.com/pandas-dev/pandas/pull/23178 is in?
Merging so we can move forward. Will keep an eye on master though.
Thanks @JustinZhengBC!