Delay import
State: closed

Closes https://github.com/pandas-dev/pandas/issues/16764

Improves performance by delaying the import of matplotlib, s3fs, pytest, and openpyxl.

Also removes our old MPL style sheet. The option was deprecated and removed, but the stylesheet hung around.

Master:

```
In [1]: %time import numpy
CPU times: user 36.3 ms, sys: 13.2 ms, total: 49.5 ms
Wall time: 68.4 ms

In [2]: %time import pandas
CPU times: user 306 ms, sys: 52.2 ms, total: 358 ms
Wall time: 392 ms
```

head:

```
In [1]: %time import numpy
CPU times: user 37.7 ms, sys: 12.4 ms, total: 50 ms
Wall time: 69 ms

In [2]: %time import pandas
CPU times: user 166 ms, sys: 40.9 ms, total: 207 ms
Wall time: 245 ms
```

I can shave off more by hardcoding `pandas.__version__`. Maybe that's worthwhile for releases?

```
In [2]: %time import pandas
CPU times: user 133 ms, sys: 30 ms, total: 163 ms
Wall time: 173 ms
```

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
I *may* have messed up some of the excel config stuff... I think all the defaults are still the same, and the tests pass, but that's a bit messy. AFACT the defaults are

- xls: xlwt (even if not installed)
- xlsm: openpyxl (even if not installed)
- xlsx: xlsxwriter if installed, else openpyxl
The next big one would be avoiding numexpr, (~21 ms), since it imports pkg_resources, which takes a while. That got a little messy though.
I'm thinking about tests for this. We should be able to whip something up with [ModuleFinder](https://docs.python.org/3.6/library/modulefinder.html)
> I can shave off more by hardcoding pandas.__version__. Maybe that's worthwhile for releases?

I don't know `versioneer` in detail - at first glance it looks more feature-rich. But we've had success with `setuptools_scm`, and it can write a file on install (https://github.com/pypa/setuptools_scm#configuration-parameters)
> I can shave off more by hardcoding pandas.__version__. Maybe that's worthwhile for releases?

NO this exactly the purpose of versioneer
to avoid this hardcoding mess we had over the years
> The next big one would be avoiding numexpr, (~21 ms), since it imports pkg_resources, which takes a while. That got a little messy though.

this could certainly be deferred till first use
> But we've had success with setuptools_scm, and it can write a file on install 

I have as well. It seems to be the "recommended" way to do automatic versioning.

> NO this exactly the purpose of versioneer

To be clear, this would just be done for releases, as part of making the release commit. Immediately after tagging, the release manager would add a commit going back to versioneer.

> this could certainly be deferred till first use

I'm trying it out now. It's not *too* bad. Will push a commit in a bit.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17710?src=pr&el=h1) Report
> Merging [#17710](https://codecov.io/gh/pandas-dev/pandas/pull/17710?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/2781b18008a7dca575a4f3496c8f11c1ea05cced?src=pr&el=desc) will **decrease** coverage by `0.04%`.
> The diff coverage is `72.28%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17710/graphs/tree.svg?src=pr&token=eZ4WkYLtcO&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/17710?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17710      +/-   ##
==========================================
- Coverage   91.25%    91.2%   -0.05%     
==========================================
  Files         163      163              
  Lines       49823    49831       +8     
==========================================
- Hits        45464    45450      -14     
- Misses       4359     4381      +22
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89% <72.28%> (-0.03%)` | :arrow_down: |
| #single | `40.26% <39.75%> (-0.13%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17710?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/17710?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [pandas/plotting/\_style.py](https://codecov.io/gh/pandas-dev/pandas/pull/17710?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19zdHlsZS5weQ==) | `74.28% <ø> (-0.25%)` | :arrow_down: |
| [pandas/core/computation/expressions.py](https://codecov.io/gh/pandas-dev/pandas/pull/17710?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vZXhwcmVzc2lvbnMucHk=) | `0% <ø> (ø)` | :arrow_up: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17710?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.73% <100%> (-0.1%)` | :arrow_down: |
| [pandas/core/config\_init.py](https://codecov.io/gh/pandas-dev/pandas/pull/17710?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29uZmlnX2luaXQucHk=) | `98.34% <100%> (+2.22%)` | :arrow_up: |
| [pandas/core/panel.py](https://codecov.io/gh/pandas-dev/pandas/pull/17710?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcGFuZWwucHk=) | `96.91% <100%> (ø)` | :arrow_up: |
| [pandas/core/internals.py](https://codecov.io/gh/pandas-dev/pandas/pull/17710?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzLnB5) | `94.37% <100%> (ø)` | :arrow_up: |
| [pandas/core/computation/eval.py](https://codecov.io/gh/pandas-dev/pandas/pull/17710?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vZXZhbC5weQ==) | `97.02% <100%> (+0.06%)` | :arrow_up: |
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/17710?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `68.64% <60%> (ø)` | :arrow_up: |
| [pandas/core/ops.py](https://codecov.io/gh/pandas-dev/pandas/pull/17710?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvb3BzLnB5) | `91.76% <75%> (-0.13%)` | :arrow_down: |
| ... and [9 more](https://codecov.io/gh/pandas-dev/pandas/pull/17710?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17710?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17710?src=pr&el=footer). Last update [2781b18...9d0f74a](https://codecov.io/gh/pandas-dev/pandas/pull/17710?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Thanks @TomAugspurger!  I'll look at the Excel stuff later, I had a half-finished branch where I think I was fighting the same config stuff you are.

I'm not sure you need to anything with the version stuff?  It is very slow on dev versions, but doesn't it get effectively hardcoded in releases already?

`~\AppData\Local\Continuum\Anaconda3\envs\py36\lib\site-packages\pandas\_version.py`

```python

# This file was generated by 'versioneer.py' (0.15) from
# revision-control system data, or from the parent directory name of an
# unpacked source archive. Distribution tarballs contain a pre-generated copy
# of this file.

from warnings import catch_warnings
with catch_warnings(record=True):
    import json
import sys

version_json = '''
{
 "dirty": false,
 "error": null,
 "full-revisionid": "3a7f956c30528736beaae5784f509a76d892e229",
 "version": "0.20.3"
}
'''  # END VERSION_JSON


def get_versions():
    return json.loads(version_json)
```


> I'm not sure you need to anything with the version stuff? It is very slow on dev versions, but doesn't it get effectively hardcoded in releases already?

Oh cool. I wasn't aware of that. With numexpr delayed:

```
In [1]: %time import numpy
CPU times: user 35.4 ms, sys: 11.1 ms, total: 46.5 ms
Wall time: 66.7 ms

In [2]: %time import pandas
CPU times: user 125 ms, sys: 24.2 ms, total: 150 ms
Wall time: 175 ms
```

Down from ~240ms
Hello @TomAugspurger! Thanks for updating the PR.







































Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on October 02, 2017 at 11:38 Hours UTC
I added a simple script in https://github.com/pandas-dev/pandas/pull/17710/files#diff-b3b48041ab2d614f95206404de64cdfe to check for certain modules whose import is triggered by pandas. This will run on Travis.
lgtm 

many run some asv (or subset) to make sure nothing changes substabtially
@TomAugspurger need to rebase I had merged something else.
@TomAugspurger lgtm. thanks!
This PR includes commit https://github.com/pandas-dev/pandas/commit/2310faa109bdfd9ff3ef4fc19a163d790d60c645 which unfortunately triggers a problem in xarray plotting, https://github.com/pydata/xarray/issues/1661 .  I don't know nearly enough about pandas to know what the right approach is to mitigate or resolve this.