cleaned up imports
State: closed

This is a continuation of #18148 and does the following

-  puts the `reshape` imports from `pandas.core.api` to `pandas.core.reshape.api`
- uses `pandas.core.dtypes.generic.ABCMultiIndex` inplace of `MultiIndex` in `melt.py`
- uses `from pandas import DataFrame` in `melt.py` inside functions
- imports directly from pandas in reshape tests

There is one test failing. Statsmodels imports `get_dummies` from `pandas.core.api` which is no longer possible. It should import directly from pandas.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Clean

Comments:
@jreback Why is it needed to use ABCMultiIndex instead of MultiIndex and move the DataFrame imports inline? There is no circular import here?

> There is one test failing. Statsmodels imports get_dummies from pandas.core.api which is no longer possible. It should import directly from pandas.

Then I think we should put it back, but we should also open an issue at statsmodels that they should not do that (independently of whether we have that import there or not, they should never import from 'core')
@jorisvandenbossche Looks like statsmodels has it fixed now. 

Edited:

@TomAugspurger fixed this here: https://github.com/statsmodels/statsmodels/commit/45e55d0180017fffd1c57e2b92143c6be0379d4d#diff-a494553e88130577065e0b012246ca51
The problem is that they didn't yet do a new release. So for now I would not do any change that breaks a released version of statsmodels, as we are not sure at this point a new statsmodel release will be out before a new pandas release.
Yes, that makes sense. But, pandas 0.22 won't come out for another 3+ months (where is the release schedule btw?) and statsmodels has been on 0.8 for 9 months now. Should we find out if they are going to release the next version before 0.22? 
Are you able to just import get_dummies in pandas.core.api, or does that
cause circular import issues? I wouldn't assume that statsmodels is going
to do a release before 0.22.

On Mon, Nov 13, 2017 at 3:53 PM, Ted Petrou <notifications@github.com>
wrote:

> Yes, that makes sense. But, pandas 0.22 won't come out for another 3+
> months (where is the release schedule btw?) and statsmodels has been on 0.8
> for 9 months now. Should we find out if they are going to release the next
> version before 0.22?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/18264#issuecomment-344070766>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIo3Lqo9WK81S6WqD38DqkHHe1HaSks5s2LpqgaJpZM4QcJx7>
> .
>

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=h1) Report
> Merging [#18264](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/7f4c960c7d0c100e5df9f0a3bc68401fc4eace4d?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/18264/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #18264      +/-   ##
==========================================
+ Coverage   91.38%   91.38%   +<.01%     
==========================================
  Files         164      164              
  Lines       49884    49884              
==========================================
+ Hits        45584    45587       +3     
+ Misses       4300     4297       -3
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.19% <100%> (+0.02%)` | :arrow_up: |
| #single | `39.41% <33.33%> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/api.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXBpLnB5) | `100% <ø> (ø)` | :arrow_up: |
| [pandas/core/reshape/melt.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9tZWx0LnB5) | `96.22% <100%> (+0.03%)` | :arrow_up: |
| [pandas/core/reshape/api.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9hcGkucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.8% <0%> (-0.1%)` | :arrow_down: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `65.25% <0%> (+1.81%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=footer). Last update [7f4c960...8782852](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

> Are you able to just import get_dummies in pandas.core.api, or does that cause circular import issues? 

That is no problem, as we were only cleaning up imports in core/api.py that are not really necessary from pandas point of view (but thus used by statsmodels). So I mean easy to just not do the clean-up (yet)
@tdpetrou 

> There is one test failing. Statsmodels imports get_dummies from pandas.core.api which is no longer possible. It should import directly from pandas.

so test_downstream.py is failing? we test against the latest released versions of these packages

I want to add a repro test with the statsmodels failure, then you can leave ``get_dummies`` import in ``pandas.core.api``, with a TODO to move later (and we'll create an issue about it).
It's been fixed in statsmodels master, just not the released version.

On Tue, Nov 14, 2017 at 5:17 AM, Jeff Reback <notifications@github.com>
wrote:

> @tdpetrou <https://github.com/tdpetrou>
>
> There is one test failing. Statsmodels imports get_dummies from
> pandas.core.api which is no longer possible. It should import directly from
> pandas.
>
> so test_downstream.py is failing?
>
> I want to add a repro test with the statsmodels failure, then you can
> leave get_dummies import in pandas.core.api, with a TODO to move later
> (and we'll create an issue about it).
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/18264#issuecomment-344227381>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIjsact-7pBl3mn2OPd30Z1I-IUhQks5s2XbDgaJpZM4QcJx7>
> .
>

@TomAugspurger that's good, but our test should still fail; its against the released version.

if its NOT failing, then our test is not good enough (maybe it doesn't import that code).
@jreback Yes it was `test_downstream.py`. Ok I'll keep it `get_dummies` import in  `pandas.core.api` with TODO

```
pandas/tests/test_downstream.py:60:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../anaconda3/lib/python3.6/site-packages/statsmodels/api.py:14: in <module>
    from .discrete.discrete_model import (Poisson, Logit, Probit,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    """
    from __future__ import division

    __all__ = ["Poisson", "Logit", "Probit", "MNLogit", "NegativeBinomial"]

    from statsmodels.compat.python import lmap, lzip, range
    import numpy as np
    from scipy.special import gammaln
    from scipy import stats, special, optimize  # opt just for nbin
    import statsmodels.tools.tools as tools
    from statsmodels.tools import data as data_tools
    from statsmodels.tools.decorators import (resettable_cache,
            cache_readonly)
    from statsmodels.regression.linear_model import OLS
    from scipy import stats, special, optimize  # opt just for nbin
    from scipy.stats import nbinom
    from statsmodels.tools.sm_exceptions import PerfectSeparationError
    from statsmodels.tools.numdiff import (approx_fprime, approx_hess,
                                           approx_hess_cs, approx_fprime_cs)
    import statsmodels.base.model as base
    from statsmodels.base.data import handle_data  # for mnlogit
    import statsmodels.regression.linear_model as lm
    import statsmodels.base.wrapper as wrap
    from statsmodels.compat.numpy import np_matrix_rank
>   from pandas.core.api import get_dummies
E   ImportError: cannot import name 'get_dummies'

../../anaconda3/lib/python3.6/site-packages/statsmodels/discrete/discrete_model.py:41: ImportError
```
@tdpetrou so this PR *should* be failing as the current code has the move of ``get_dummies``. yes I want to leave it until its fixed in statsmodels, but why does this NOT fail? this would indicate the test is flawed.
@jreback `test_downstream.py` is failing on my machine. Are you implying that the CI services are linking to the unreleased statsmodels versions?
this is puzzling why this is not failing, going to test on another build, #18333 let's see.
@tdpetrou go ahead and rebase, this *should* fail. 
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=h1) Report
> Merging [#18264](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/cfad581e9a6d35c7d05d2b1f34e4a19f7ee15cc6?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/18264/graphs/tree.svg?src=pr&token=eZ4WkYLtcO&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #18264      +/-   ##
==========================================
+ Coverage   91.38%   91.38%   +<.01%     
==========================================
  Files         164      164              
  Lines       49790    49884      +94     
==========================================
+ Hits        45501    45587      +86     
- Misses       4289     4297       +8
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.19% <100%> (+0.02%)` | :arrow_up: |
| #single | `39.41% <33.33%> (-0.15%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/api.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXBpLnB5) | `100% <ø> (ø)` | :arrow_up: |
| [pandas/core/reshape/melt.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9tZWx0LnB5) | `96.22% <100%> (+0.03%)` | :arrow_up: |
| [pandas/core/reshape/api.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9hcGkucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/util/\_decorators.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX2RlY29yYXRvcnMucHk=) | `80.7% <0%> (-0.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.8% <0%> (-0.1%)` | :arrow_down: |
| [pandas/core/sparse/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL3Nlcmllcy5weQ==) | `95.28% <0%> (-0.06%)` | :arrow_down: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `95% <0%> (-0.02%)` | :arrow_down: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `95.72% <0%> (-0.01%)` | :arrow_down: |
| [pandas/core/reshape/reshape.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9yZXNoYXBlLnB5) | `100% <0%> (ø)` | :arrow_up: |
| ... and [15 more](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=footer). Last update [cfad581...794dd6d](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=h1) Report
> Merging [#18264](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/dac9b43f7c12b3785250b0e7805b43421a03dc8c?src=pr&el=desc) will **increase** coverage by `0.29%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/18264/graphs/tree.svg?src=pr&token=eZ4WkYLtcO&width=650&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #18264      +/-   ##
==========================================
+ Coverage   91.35%   91.65%   +0.29%     
==========================================
  Files         163      164       +1     
  Lines       49681    51487    +1806     
==========================================
+ Hits        45386    47189    +1803     
- Misses       4295     4298       +3
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.52% <100%> (+0.39%)` | :arrow_up: |
| #single | `40.54% <33.33%> (+0.8%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/api.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXBpLnB5) | `100% <ø> (ø)` | :arrow_up: |
| [pandas/core/reshape/melt.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9tZWx0LnB5) | `96.22% <100%> (+0.03%)` | :arrow_up: |
| [pandas/core/reshape/api.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9hcGkucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/tseries/offsets.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvb2Zmc2V0cy5weQ==) | `96.7% <0%> (-0.31%)` | :arrow_down: |
| [pandas/core/resample.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzYW1wbGUucHk=) | `96.16% <0%> (-0.19%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.8% <0%> (-0.1%)` | :arrow_down: |
| [pandas/compat/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9fX2luaXRfXy5weQ==) | `58.1% <0%> (-0.09%)` | :arrow_down: |
| [pandas/core/indexes/timedeltas.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy90aW1lZGVsdGFzLnB5) | `91.14% <0%> (-0.04%)` | :arrow_down: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `95.73% <0%> (ø)` | :arrow_up: |
| ... and [9 more](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=footer). Last update [dac9b43...1f56dfd](https://codecov.io/gh/pandas-dev/pandas/pull/18264?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

can you rebase 1 more time, I want to see this fail; the statsmodels test was being skipped.
@jreback Did this fail correctly?
Yes, it is correctly failing now. So you can now put it back in core/api.py

@jreback can you answer my question above https://github.com/pandas-dev/pandas/pull/18264#issuecomment-344061156 ?
@jorisvandenbossche I put `get_dummies` back in `core.api`. I left the `get_dummies` import in `core.reshape.api`.
> Why is it needed to use ABCMultiIndex instead of MultiIndex and move the DataFrame imports inline? There is no circular import here?

we use the ABC* for instance checking generally, these have no dependencies. This actually *does* have a conditional import, that's why its pandas.core.frame.DataFrame; we try to remove these and import inside the function as its much cleaner.

@tdpetrou 

> I put get_dummies back in core.api. I left the get_dummies import in core.reshape.api.

thanks. ping on green.

Can you make an issue to move this (on the pandas side, when statsmodels releases).
@jreback I think this is good but circleci is failing?

I'll keep a note to make an issue on the next statsmodels release
thanks @tdpetrou 

> I'll keep a note to make an issue on the next statsmodels release

can you make an issue about this, we have 2 changes we need to make now (I'll edit with the other one)