Remove py27 CI jobs
State: closed

The start of a looooooooong process to get rid of the py2 compat code.

I kept most of the CI jobs that used to be 2.7 at the lower end of the various supported versions, with 2x 2.7->3.5, 1x 2.7->3.6 and only one real upgrade from 2.7->3.7 (first CI job for 3.7 & windows).

@jreback @TomAugspurger 


Labels:
name: Build
description: Library building on various platforms

Comments:
keep in mind that we likely won't merge anything PY2 removal for a month or so.
> keep in mind that we likely won't merge anything PY2 removal for a month or so.

Which makes a lot of sense from a backports POV, but I didn't think about that. Guess I just couldn't/can't wait. ;)
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24942?src=pr&el=h1) Report
> Merging [#24942](https://codecov.io/gh/pandas-dev/pandas/pull/24942?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/bb43726e1f52a0ddee45fcf485690719f262870d?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24942/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24942?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24942      +/-   ##
==========================================
+ Coverage   92.37%   92.37%   +<.01%     
==========================================
  Files         166      166              
  Lines       52420    52388      -32     
==========================================
- Hits        48423    48396      -27     
+ Misses       3997     3992       -5
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.8% <ø> (ø)` | :arrow_up: |
| #single | `42.9% <ø> (+0.01%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24942?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/clipboard/clipboards.py](https://codecov.io/gh/pandas-dev/pandas/pull/24942/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NsaXBib2FyZC9jbGlwYm9hcmRzLnB5) | `28.23% <0%> (-2.36%)` | :arrow_down: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/24942/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.32% <0%> (-0.23%)` | :arrow_down: |
| [pandas/core/reshape/merge.py](https://codecov.io/gh/pandas-dev/pandas/pull/24942/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9tZXJnZS5weQ==) | `94.4% <0%> (-0.08%)` | :arrow_down: |
| [pandas/core/indexes/multi.py](https://codecov.io/gh/pandas-dev/pandas/pull/24942/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9tdWx0aS5weQ==) | `95.59% <0%> (-0.02%)` | :arrow_down: |
| [pandas/core/indexes/range.py](https://codecov.io/gh/pandas-dev/pandas/pull/24942/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9yYW5nZS5weQ==) | `97.36% <0%> (-0.01%)` | :arrow_down: |
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/24942/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `96.27% <0%> (-0.01%)` | :arrow_down: |
| [pandas/core/resample.py](https://codecov.io/gh/pandas-dev/pandas/pull/24942/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzYW1wbGUucHk=) | `97.23% <0%> (-0.01%)` | :arrow_down: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/24942/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.63% <0%> (-0.01%)` | :arrow_down: |
| [pandas/core/tools/timedeltas.py](https://codecov.io/gh/pandas-dev/pandas/pull/24942/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdG9vbHMvdGltZWRlbHRhcy5weQ==) | `98% <0%> (ø)` | :arrow_up: |
| [pandas/core/tools/numeric.py](https://codecov.io/gh/pandas-dev/pandas/pull/24942/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdG9vbHMvbnVtZXJpYy5weQ==) | `100% <0%> (ø)` | :arrow_up: |
| ... and [7 more](https://codecov.io/gh/pandas-dev/pandas/pull/24942/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24942?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24942?src=pr&el=footer). Last update [bb43726...b88b87d](https://codecov.io/gh/pandas-dev/pandas/pull/24942?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24942?src=pr&el=h1) Report
> Merging [#24942](https://codecov.io/gh/pandas-dev/pandas/pull/24942?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/a61d8232c0293ba21c8c881c3959a3eaade19afc?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24942/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24942?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24942      +/-   ##
==========================================
- Coverage   91.25%   91.25%   -0.01%     
==========================================
  Files         172      172              
  Lines       52976    52976              
==========================================
- Hits        48342    48341       -1     
- Misses       4634     4635       +1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.82% <ø> (ø)` | :arrow_up: |
| #single | `41.74% <ø> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24942?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/24942/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `88.98% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24942?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24942?src=pr&el=footer). Last update [a61d823...fe83f86](https://codecov.io/gh/pandas-dev/pandas/pull/24942?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@TomAugspurger @jorisvandenbossche @jreback 

First passing commit here. Might still need a bit of work for the weird SQL/excel failures, but a few PY35-only failures don't seem so critical IMO (dropping 3.5 when 3.8 is released at the end of 2019?).
Hello @h-vetinari! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-03-15 15:17:33 UTC
Finally back to green...
Added some more fixes in `install.rst`.
this will have to wait for #25554 (which needs to wait for 0.24.2 release)
Failure of travis doc job seems to be due to jupyter/notebook#4311.
@h-vetinari ok, let's work on getting this in a good state; will merge before the numpy bump. THEN we can change actual code.
eg.
```
setup(name=DISTNAME,
      maintainer=AUTHOR,
      version=versioneer.get_version(),
      packages=find_packages(include=['pandas', 'pandas.*']),
      package_data={'': ['templates/*', '_libs/*.dll']},
      ext_modules=maybe_cythonize(extensions, compiler_directives=directives),
      maintainer_email=EMAIL,
      description=DESCRIPTION,
      license=LICENSE,
      cmdclass=cmdclass,
      url=URL,
      download_url=DOWNLOAD_URL,
      long_description=LONG_DESCRIPTION,
      classifiers=CLASSIFIERS,
      platforms='any',
      python_requires='>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*',
      **setuptools_kwargs)
```
Matplotlib includes this: https://github.com/matplotlib/matplotlib/blob/f9a63d77b7d17afbc7eba4c30e12e5e0888c5115/setup.py#L6-L21 as a check, for older versions of pip / setuptools.

No preference on whether we do that in this PR or a followup.
@TomAugspurger what I meant was it needs to be updated, e.g. 

``python_requires='>=3.5.*` i think might work
Right. It's also recommended to include something like MPL's code for older
versions of pip / setuptools that don't honor python_requires:
https://python3statement.org/practicalities/

On Thu, Mar 14, 2019 at 8:08 AM Jeff Reback <notifications@github.com>
wrote:

> @TomAugspurger <https://github.com/TomAugspurger> what I meant was it
> needs to be updated, e.g.
>
> ``python_requires='>=3.5.*` i think might work
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/24942#issuecomment-472845440>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIgHKdJ0z-D2P6yOK6h1yFnAax7l-ks5vWknNgaJpZM4aT3yR>
> .
>

@jreback @TomAugspurger 
The last couple of builds have had weird (but consistent) travis errors in the `travis_35` job, where the worker nodes crash after about ~10 sec (so I'm thinking it's not related to actual tests segfaulting, but somehow the interplay between different versions of "infrastructure packages).
@h-vetinari can you point to the test errors for which you needed to add the additional xfails? (or do a new build without any of them) Ideally, we should investigate each of those in more detail why they are failing instead of just xfailing them (unless you already know the reason for each of them)
@jorisvandenbossche 
The fails are distributed over the CI runs throughout the thread, but I've reverted them in the last commit (including the regex fixes @jreback wanted me to revert), so then there's a single place to inspect the failures. Also - who knows - maybe some of them resolved themselves...
@jorisvandenbossche @TomAugspurger @jreback 
Please check out https://travis-ci.org/pandas-dev/pandas/jobs/506535600, it fails already in the `single` stage with the tests I would have xfailed. When I first started this PR I played around with bumping the versions of xlrd etc (even when unpinning `pymysql/xlsxwriter`), but that didn't solve things either.

Considering the fact that removing support for PY35 is already being talked about in #25725, I think a small handful of xfailed tests is not a big burden. Certainly those are less trivial to fix than for example the skips that have been added for regex fails (of which there about 30).
as i said above pls start over and JUST remove the 2.7 jobs; since things are working now this will just work. after that you can do independent PR to add new / change things.

something is breaking and certainly needs fixing
but not here
PY35 is not being removed at this time
> @jreback: as i said above pls start over and JUST remove the 2.7 jobs; since things are working now this will just work. after that you can do independent PR to add new / change things.

Of course it's easier to just rip things out, but I actually went to the effort here of thinking of a good distribution for the CI, updating the jobs, and making them pass again.

That's also how the last numpy bump was made. Tbh, I feel these conditions are quite arbitrary, but well... I'll delete the former PY2 jobs later today, and post the fixes in a separate PR. 

> PY35 is not being removed at this time

I know that. But adding 6-7 `PY35`-specific xfails (for non-core functionality) aren't that severe IMO, considering the point in the lifecycle.
@h-vetinari as I have explained many many times. PRs which do lots of things take *way* more time to review and are much harder to merge. It is the same case here. 
@jreback 
This is green now. If only it always were as easy as deleting CI jobs, haha.
thanks, would also appreciate a PR against : https://github.com/Macpython/pandas-wheels (the daily branch, not master).
> thanks, would also appreciate a PR against : https://github.com/Macpython/pandas-wheels (the daily branch, not master).

This same PR or the PY35 one?

neither. one to remove the PY2 jobs only (this is where we build wheels), and is our daily testing.