DEPS: Bump numpy to 1.13.3
State: closed

#25227 is trying to add `pyproject.toml` which is necessary for the whole `pip`-machinery to work smoothly post v.19.0. However, this seems to be running into problems with a numpy version bump, and mood there was to bump numpy requirement to 1.13.x.

The bare minimum for the bump is implemented in that PR, but a version bump requires getting rid of a bunch of compat code (and CI adjustments), otherwise the cruft just accumulates (it took me three PRs to clean out until 1.12).

Scipy is goint to [bump to 1.13.3](https://github.com/pandas-dev/pandas/pull/25227#issuecomment-464128057) with the next release (due to cython issues, I believe), so I thought I'll directly take it to 1.13.3 here.

I tried redistributing the CI jobs as best as I could. Here's the before/after:

**EDIT: updated pins in https://github.com/pandas-dev/pandas/pull/25554#issuecomment-475881206**

CI job: line number | before | this PR
-------|-----------------------------|-------------------
`ci/deps/azure-27-compat.yaml:10:` | `numpy=1.12.0` | `numpy=1.13.3`
`ci/deps/azure-27-locale.yaml:10:` | `numpy=1.12.0` |  `numpy=1.14.*`
`ci/deps/azure-36-locale_slow.yaml:16:` | `numpy` | `numpy=1.15.*`
`ci/deps/azure-37-locale.yaml:15:` | `numpy` | `numpy`
`ci/deps/azure-37-numpydev.yaml:18:` | `numpy` | `numpy`
`ci/deps/azure-macos-35.yaml:14:` | `numpy=1.12.0` | `numpy=1.13.3`
`ci/deps/azure-windows-27.yaml:15:` | `numpy=1.12*` | `numpy=1.13.3`
`ci/deps/azure-windows-36.yaml:12:` | `numpy=1.14*` | `numpy=1.15.*`
`ci/deps/travis-27.yaml:20:` | `numpy=1.13*` | `numpy=1.14.*`
`ci/deps/travis-36-doc.yaml:23:` | `numpy=1.13*` | `numpy`
`ci/deps/travis-36-locale.yaml:15:` | `numpy` | `numpy`
`ci/deps/travis-36-slow.yaml:12:` | `numpy` | `numpy`
`ci/deps/travis-36.yaml:17:` | `numpy` | `numpy=1.15.*`
`ci/deps/travis-37.yaml:10:` | `numpy` | `numpy`

Matplotlib needs to be bumped as well (since current min 2.0.0 cannot be resolved by conda together with numpy 1.13.3), but it's only a tiny bump to ~2.0.2~ ~2.1.0~ ~2.1.1~ 2.2.2. In any case, the version spread should probably start to reflect that there's a bunch of mpl 3.0.x about, and so I also changed the spread a bit here:

CI job: line number | before | this PR
-------------|-------|-----------
`ci/deps/azure-27-locale.yaml:9:` | `matplotlib=2.0.0` | `matplotlib=2.2.2`
`ci/deps/azure-36-locale_slow.yaml:13:` | `matplotlib` | `matplotlib=3.0.*`
`ci/deps/azure-37-locale.yaml:12:` | `matplotlib` | `matplotlib`
`ci/deps/azure-macos-35.yaml:11:` | `matplotlib=2.2.0` | `matplotlib=2.2.3`
`ci/deps/azure-windows-27.yaml:13:` | `matplotlib=2.0.1` | `matplotlib=2.2.4`
`ci/deps/azure-windows-36.yaml:10:` | `matplotlib` | `matplotlib`
`ci/deps/travis-27.yaml:16:` | `matplotlib=2.2.2` | `matplotlib=2.2.*`
`ci/deps/travis-36-doc.yaml:17:` | `matplotlib` | `matplotlib`
`ci/deps/travis-36-locale.yaml:12:` | `matplotlib` | `matplotlib=3.0.1`
`ci/deps/travis-36-slow.yaml:10:` | `matplotlib` | `matplotlib`
`ci/deps/travis-36.yaml:14:` | `matplotlib` | `matplotlib=3.0.0`

It's probably wishful thinking that everything will pass straight away (especially when mpl versions change, see the code in `pandas/tests/plotting/test_datetimelike.py`), but oh well...


Labels:
name: Dependencies
description: Required and optional dependencies

Comments:
> Scipy is goint to bump to 1.13.3 with the next release (due to cython issues, I believe), so I thought I'll directly take it to 1.13.3 here.

Not due to Cython issues, just to reduce the maintenance burden. See https://mail.python.org/pipermail/scipy-dev/2018-November/023181.html for rationale.

+1 for going to 1.13.3 as well. Matplotlib is doing a similar thing (actually they want to be a bit more agressive), and wants to standardize a common Python/NumPy version support policy for core packages (see https://github.com/scipy/scipy/issues/9610, we'll come back on that soon).
`install.rst` also needs a small update:

https://github.com/pandas-dev/pandas/blob/615fbb3ce4bb171e8610c590a9f9899c60028d54/doc/source/install.rst#L227
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25554?src=pr&el=h1) Report
> Merging [#25554](https://codecov.io/gh/pandas-dev/pandas/pull/25554?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d10bbce349b293e9f753d799cd2ad032d5ec7333?src=pr&el=desc) will **decrease** coverage by `0.48%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25554/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25554?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25554      +/-   ##
==========================================
- Coverage   91.75%   91.26%   -0.49%     
==========================================
  Files         173      173              
  Lines       52960    52961       +1     
==========================================
- Hits        48595    48337     -258     
- Misses       4365     4624     +259
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.83% <100%> (-0.51%)` | :arrow_down: |
| #single | `41.73% <75%> (+0.01%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25554?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `95.96% <ø> (ø)` | :arrow_up: |
| [pandas/core/arrays/sparse.py](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3NwYXJzZS5weQ==) | `92.17% <ø> (ø)` | :arrow_up: |
| [pandas/compat/numpy/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9udW1weS9fX2luaXRfXy5weQ==) | `93.1% <100%> (-0.23%)` | :arrow_down: |
| [pandas/plotting/\_compat.py](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb21wYXQucHk=) | `87.5% <100%> (ø)` | :arrow_up: |
| [pandas/core/groupby/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9nZW5lcmljLnB5) | `86.98% <100%> (-0.02%)` | :arrow_down: |
| [pandas/core/arrays/numpy\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL251bXB5Xy5weQ==) | `94.49% <100%> (+0.83%)` | :arrow_up: |
| [pandas/core/panel.py](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcGFuZWwucHk=) | `38.56% <0%> (-33.19%)` | :arrow_down: |
| [pandas/core/sparse/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL3Nlcmllcy5weQ==) | `93.3% <0%> (-2.24%)` | :arrow_down: |
| [pandas/core/indexing.py](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhpbmcucHk=) | `90.88% <0%> (-1.41%)` | :arrow_down: |
| [pandas/core/internals/managers.py](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL21hbmFnZXJzLnB5) | `93.92% <0%> (-0.92%)` | :arrow_down: |
| ... and [5 more](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25554?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25554?src=pr&el=footer). Last update [d10bbce...5b22257](https://codecov.io/gh/pandas-dev/pandas/pull/25554?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25554?src=pr&el=h1) Report
> Merging [#25554](https://codecov.io/gh/pandas-dev/pandas/pull/25554?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/882961df6025b800b899d351c243412e799accf4?src=pr&el=desc) will **increase** coverage by `0.02%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25554/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25554?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25554      +/-   ##
==========================================
+ Coverage   91.53%   91.55%   +0.02%     
==========================================
  Files         175      175              
  Lines       52808    52775      -33     
==========================================
- Hits        48338    48320      -18     
+ Misses       4470     4455      -15
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.12% <100%> (+0.02%)` | :arrow_up: |
| #single | `41.82% <71.42%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25554?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/missing.py](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvbWlzc2luZy5weQ==) | `93.96% <ø> (+1.39%)` | :arrow_up: |
| [pandas/util/\_test\_decorators.py](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX3Rlc3RfZGVjb3JhdG9ycy5weQ==) | `93.22% <ø> (+0.57%)` | :arrow_up: |
| [pandas/core/arrays/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2RhdGV0aW1lcy5weQ==) | `97.79% <ø> (-0.01%)` | :arrow_down: |
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `95.96% <ø> (ø)` | :arrow_up: |
| [pandas/core/arrays/sparse.py](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3NwYXJzZS5weQ==) | `92.17% <ø> (ø)` | :arrow_up: |
| [pandas/core/computation/check.py](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vY2hlY2sucHk=) | `92.3% <100%> (ø)` | :arrow_up: |
| [pandas/core/nanops.py](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvbmFub3BzLnB5) | `93.84% <100%> (ø)` | :arrow_up: |
| [pandas/core/arrays/numpy\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL251bXB5Xy5weQ==) | `94.49% <100%> (+0.83%)` | :arrow_up: |
| [pandas/compat/numpy/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9udW1weS9fX2luaXRfXy5weQ==) | `93.1% <100%> (-0.23%)` | :arrow_down: |
| [pandas/core/groupby/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9nZW5lcmljLnB5) | `87.03% <100%> (-0.02%)` | :arrow_down: |
| ... and [4 more](https://codecov.io/gh/pandas-dev/pandas/pull/25554/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25554?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25554?src=pr&el=footer). Last update [882961d...20ad5a5](https://codecov.io/gh/pandas-dev/pandas/pull/25554?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

I like @h-vetinari's proposed pins in 
https://github.com/pandas-dev/pandas/pull/25554#issue-258441536
@jreback @TomAugspurger 
This should be done now. :)
@TomAugspurger 
That sounds like a separate PR?

@jreback 
PTAL
> > @TomAugspurger: @h-vetinari if you're interested [...]

> @h-vetinari: @TomAugspurger That sounds like a separate PR?

I've got an initial draft prepared, but will wait until this one is merged.
this needs to wait until 0.24.2 is done and needs review
@jreback 
Checked the MPL releases, and `2.2.2` is barely a year old. Therefore, I bumped only to `2.1.0`.
@jreback 
OK, nevermind, I went to matplotlib 2.2.2 after all. By the time pandas 0.24.2 hits, this will be over a year old, and there's just too much variability in the MPL point releases for `2.1.x` so that any starting point in `2.0.1 | 2.0.2 | 2.1.0 | 2.1.1 | 2.1.2` (and to a degree `2.2.0`) would lead to lots of compat code (along the lines of what you didn't like above). And since 2.2.2 was only released 10 days after 2.2.0, I went to 2.2.2 right away.
Sorry for the Matplotlib API thrashing.  

2.2.x is the LTS version so it is a good choice!
@jreback 
I keep running into a weird cython error on the `azure-windows-27` job:
```
running build_ext
pandas._libs.algos: -> [['pandas\\_libs/algos.c']]
Traceback (most recent call last):
  File "setup.py", line 751, in <module>
    **setuptools_kwargs)
  File "C:\Miniconda\lib\site-packages\setuptools\__init__.py", line 143, in setup
    return distutils.core.setup(**attrs)
  File "C:\Miniconda\lib\distutils\core.py", line 148, in setup
    dist.run_commands()
  File "C:\Miniconda\lib\distutils\dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "C:\Miniconda\lib\distutils\dist.py", line 985, in run_command
    cmd_obj.run()
  File "C:\Miniconda\lib\distutils\command\build_ext.py", line 339, in run
    self.build_extensions()
  File "setup.py", line 372, in build_extensions
    self.check_cython_extensions(self.extensions)
  File "setup.py", line 369, in check_cython_extensions
    """.format(src=src))
Exception: Cython-generated file 'pandas\_libs/algos.c' not found.
                Cython is required to compile pandas from a development branch.
                Please install Cython or download a release package of pandas.
```

Bumping cython didn't help (resp. then the 3.5 jobs can't satisfy min). How about we drop the PY2 CI (#24942) first before bumping numpy?
@h-vetinari can you merge master on this now that Py2 CI jobs are out?
> @WillAyd: @h-vetinari can you merge master on this now that Py2 CI jobs are out?

Since #24942 was split up, this PR here still depends on #25752. It would be helpful if you could give the latter one a look. :)
Currently rebasing on top of #25752. This is the current set of pins for python/numpy/matplotlib. Comments welcome.

CI job: line number | OS | python | numpy | mpl
-------|-----------------------------|--------|----------|---
`azure-35-compat.yaml:10:` | posix | `python=3.5.*` | `numpy=1.13.3` | -
`azure-36-locale.yaml:10:` | posix | `python=3.6.*` | `numpy=1.14.*` | `matplotlib=2.2.2`
`azure-36-locale_slow.yaml:16:` | posix |  `python=3.6.*` | `numpy=1.15.*`  | `matplotlib=3.0.*` 
`azure-37-locale.yaml:15:` | posix |  `python=3.7.*` | `numpy`  | `matplotlib`
`azure-37-numpydev.yaml:18:` | posix |  `python=3.7.*` | `numpy`  | -
`azure-macos-35.yaml:14:` | macos |  `python=3.5.*` | `numpy=1.13.3` | `matplotlib=2.2.3`
`azure-windows-36.yaml:15:` | win |  `python=3.6.*` | `numpy=1.15.*` | `matplotlib=3.0.2`
`azure-windows-37.yaml:12:` | win |  `python=3.7.*` | `numpy=1.14.*` | `matplotlib=2.2.*`
`travis-36-cov.yaml:17:` | posix | `python=3.6.*` | `numpy=1.15.*` | `matplotlib`
`travis-36-doc.yaml:23:` | posix |  `python=3.6.*` | `numpy` | `matplotlib`
`travis-36-locale.yaml:15:` | posix | `python=3.6.*` | `numpy` | `matplotlib=3.0.0`
`travis-36-slow.yaml:12:` | posix | `python=3.6.*` | `numpy` | `matplotlib`
`travis-37.yaml:10:` | posix | `python=3.7.*` | `numpy` | -

PS. Note that matplotlib `3.0.1` was superseded by `3.0.2` within a day.
see if we can remove 1 or more jobs keeping the same coverage (reasonably) of deps (can be a follow up)
i don’t think it’s a big deal if we only test oldest numpy and then newest
1.14/1.15 or pretty much compat with 1.16
Well, I didn't design the CI jobs - there's OSes, numpies, locales, DB tests. Not sure we should reduce much more, but from the table above, I'd get rid of `travis-3.6-locale`.

Maybe @tomaugspurger @jorisvandenbossche want to opine?

Also, we still need to actually merge #25752.
https://travis-ci.org/pandas-dev/pandas/jobs/510922188 appears to be failing
@jreback 
Argh, now we're right back to the errors I had fixed with the regex stuff that you reverted against my request. #25874
> It will not fail the wheel building CI. I actually checked the wheel builder failures from #25867 before giving my review in #25874.

not sure what you are talking about. it DID fail. The reverted does not. If you fixed it great. I want to avoid having broken CI's endlessly.
thanks
@h-vetinari looks like this broke the gbq tests; these are pretty funky and I don't think are tested on  actual branches (thoguh they used to be AFAIR): see https://travis-ci.org/pandas-dev/pandas/jobs/512494733

so pls do a PR to disable for now.
> @jreback: not sure what you are talking about. it DID fail. The reverted does not. If you fixed it great. I want to avoid having broken CI's endlessly.

I did fail, and I had fixed it (not by reverting exactly, but then I've tried to explain that already twice). Now it's more simplified anyway, which is IMO anyway a better approach for matching these error messages.