CLN: clean-up internal sparse imports + restructure sparse submodule
State: closed

Updating our internal imports to use the path were they actually live, not the old `core.sparse.api` one.

While doing this, I was thinking now that for the tests, we could actually also use the top-level `pd.` imports.

@TomAugspurger in the `core/sparse/` module, the only actual code living there is the scipy_sparse code. I moved that to `arrays` module (but could also be moved within the sparse.py file, or create a sparse subdirectory, ..). 

Do we want to raise a deprecation warning on pandas.core.sparse.api?

Labels:
name: Sparse
description: Sparse Data Type
name: Clean

Comments:
Thanks! I had two other followup items:

1. I forgot to add a note to 1.0.0.rst that DataFrame.to_sparse was removed
2. There's a file `pandas/tests/sparse/test_combine_concat.py` that can be moved to `pandas/tests/arrays/sparse`

Do you want to do those here?

> Do we want to raise a deprecation warning on pandas.core.sparse.api?

Either doing it now, or as part of a larger `pandas.core` deprecation sounds fine to me.
Does scipy_sparse belong in core.arrays?  I've been thinking that SparseAccessor might not belong in the array file, could go in core.sparse
How about making `pandas/core/arrays/sparse` a directory? And then splitting into

- /sparse/__init__.py
- /sparse/dtype.py
- /sparse/array_.py
- /sparse/acessor.py
- /sparse/scipy_.py

It is correct that the accessor (which also is a DataFrame accessor) is not directly related to arrays, and more something on the "dataframe level". But since we do not have a proper home right now for such functionality, I think keeping the type-specific accessors close to their type implementations (array / dtype) makes somewhat sense. 

Given that, having a sparse directory with several files seems fine to me.

> I think keeping the type-specific accessors close to their type implementations (array / dtype) makes somewhat sense.

FWIW, I slightly prefer having the accessor close to the array implementation. This more closely mirrors what 3rd party EA authors would do.
> FWIW, I slightly prefer having the accessor close to the array implementation. This more closely mirrors what 3rd party EA authors would do.

This isn't a particularly big deal, but I want to make sure my reasoning is written down somewhere:

For the most part, a reader can understand `core.arrays` without needing to understand the rest of `core` (except for `core.dtypes`).  I view this as a desirable characteristic, the more de-coupling the better.

In the case of `SparseAccessor`, I know that nothing else in `core.arrays` depends on it so I can just pretend it isn't there when trying to reason about the arrays in isolation.  But to a new reader that might not be immediately obvious.  So that is both a) not a big deal and b) not a pattern I want to encourage.

@jbrockmendel and is the alternative that you prefer having a `pandas/core/sparse` with only the accessor? Or gather them in `pandas/core/accessors` or something?

For having it in arrays/sparse, the only additional thing that it relies upon from the rest of core is `PandasDelegate`, so it is not that it brings in a lot of the rest of core. And on the other hand, to understand the SparseAccessor, you need to understand the SparseArray, as is mostly just delegates methods to the SparseArray.
> and is the alternative that you prefer having a pandas/core/sparse with only the accessor? Or gather them in pandas/core/accessors or something?

At the time I formed this preference, sparse.frame and sparse.series still existed, so moving to sparse.accessor _was_ my preferred alternative.  Haven't thought about an updated preference.

> the only additional thing that it relies upon from the rest of core is `PandasDelegate`

There are runtime imports of DataFrame, Series, Index.
>  I had two other followup items:

Done

> How about making pandas/core/arrays/sparse a directory?

Went ahead with that then. 
@TomAugspurger @jbrockmendel OK?
lgtm