Use conda-env for CI environments
State: closed

Closes https://github.com/pandas-dev/pandas/issues/20720

The old setup where things were installed in multiple places resulted in some strange environments, with multiple copies of the same package in `site-packages`.

This attempts to reduce the environment definition to a single file per CI job, and the environment installation to 3 steps

- create the environment
- uninstall pandas if it's present (installing a pandas dependency)
- build pandas from source & install

Travis only for now. Will move over the appveyor / circle ones too.

Labels:
name: CI
description: Continuous Integration

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20767?src=pr&el=h1) Report
> Merging [#20767](https://codecov.io/gh/pandas-dev/pandas/pull/20767?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/0ae7e9090cee6e10ebb4124a0b7f3b30811ccc53?src=pr&el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/20767/graphs/tree.svg?src=pr&height=150&token=eZ4WkYLtcO&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/20767?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #20767      +/-   ##
==========================================
- Coverage   91.85%   91.81%   -0.04%     
==========================================
  Files         153      153              
  Lines       49308    49310       +2     
==========================================
- Hits        45290    45274      -16     
- Misses       4018     4036      +18
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.2% <ø> (-0.04%)` | :arrow_down: |
| #single | `41.87% <ø> (-0.02%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/20767?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/20767/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `65.07% <0%> (-1.74%)` | :arrow_down: |
| [pandas/core/nanops.py](https://codecov.io/gh/pandas-dev/pandas/pull/20767/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvbmFub3BzLnB5) | `95.13% <0%> (-1.17%)` | :arrow_down: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/20767/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.68% <0%> (ø)` | :arrow_up: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/20767/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.17% <0%> (ø)` | :arrow_up: |
| [pandas/core/accessor.py](https://codecov.io/gh/pandas-dev/pandas/pull/20767/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYWNjZXNzb3IucHk=) | `98.71% <0%> (+0.01%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20767?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20767?src=pr&el=footer). Last update [0ae7e90...8b0e277](https://codecov.io/gh/pandas-dev/pandas/pull/20767?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Just one failure, on the OSX build. That's pinned to numpy 1.10.4, which forces fastparquet 0.1.3, which is not compatible with thrift 0.11. I could pin fastparquet & thrift to those versions, but I think fastparquet is adequately tested elsewhere, so I've removed it.
Updated for circle-ci and appveyor.

AFAICT, the `requirements-3.5.*` files are unused. Does that look correct?
OK, passing now.

cc @jreback @jorisvandenbossche.

I verified that that Travis tests all have the same number of skips / passes. The only difference is https://travis-ci.org/pandas-dev/pandas/jobs/369333014#L2982, which apparently isn't run on PRs.
conda-forge is only used for packages that are not found in the main
channel. The 3.5-osx env uses s3fs, moto, and feather which aren't in main.

On Sat, Apr 21, 2018 at 4:58 PM, Jeff Reback <notifications@github.com>
wrote:

> *@jreback* commented on this pull request.
> ------------------------------
>
> In ci/travis-27.yaml
> <https://github.com/pandas-dev/pandas/pull/20767#discussion_r183220265>:
>
> > @@ -0,0 +1,49 @@
> +name: pandas
> +channels:
>
> can we have one of the main builds NOT use conda-forge (maybe the 3.5
> one)? in the original we had *only* the 3.6 build using it.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/20767#pullrequestreview-114184235>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIm1m0wI5okUh48J5wdN242UHGUIbks5tq6uFgaJpZM4Tdo8r>
> .
>

> conda-forge is only used for packages that are not found in the main
channel. The 3.5-osx env uses s3fs, moto, and feather which aren't in main.

I understand. my point is in the existing setup, we install from defaults ONLY for most packages. I am loath to give that up on at least 1 build. I would either go back to 2 steps on 1 or more builds, or remove these deps (e.g. you can install from pip instead these packages).
I think you can completely remove the JOB= env variable as well here (or maybe rename ENV_FILE to JOB)
also I don't think the issue number is correct here, #20720 ?
> you can install from pip instead these packages

I'd like to avoid installing moto from pip if possible. It brings in a lot of dependencies, which aren't managed by conda.

This closes #20720. It looks like that was caused by the 2 / 3 stage installs, since we ended up with multiple versions of requests, boto, etc., in the same environment. Now we just have the one. I haven't seen any of the network failures on this branch.
ok I recently added some xfails for #20720, can you remove in this PR?
Does the `conda-forge::packgage` syntax work for environment files? 
That might be another way to install some packages from conda-forge in an environment without adding conda-forge as a full channel (as then it might be more difficult to control which ones are installed from conda-forge / defaults)
Small note: I personally liked the JOB tags to have a quick overview of the different build types. I know the ENV_FILE tag is similarly unique, but a bit less user friendly to read or to refer to ("check travis job ...")
`Does the conda-forge::packgage syntax work for environment files?`

Not correctly unfortunately :( When I tried that, all the packages came from conda-forge. I suspect conda parses the deps for channels, and if found adds them to the start of the package list.  Opened https://github.com/conda/conda/issues/7202 for that.
Reverted the `JOB` removal.
> Small note: I personally liked the JOB tags to have a quick overview of the different build types. I know the ENV_FILE tag is similarly unique, but a bit less user friendly to read or to refer to ("check travis job ...")

I don't have a problem with the slightly longer ENV_FILE, maybe just rename this to JOB?
having the duplicated ones is a bit annoying
Or maybe like before where the JOB tag is the variable part in the file name? 
By file name, do you mean the environment.yaml? Because if so, I think
that's the case.

I don't really care what we do here. In general, I like the idea of a small
tag that identifies the relevant bit for each build. Now that we aren't
using the `{.build,run,pip,sh}` files, we're free to put whatever we want
in `JOB`.

I removed the unused 3.5 files.

On Mon, Apr 23, 2018 at 8:42 AM, Joris Van den Bossche <
notifications@github.com> wrote:

> Or maybe like before where the JOB tag is the variable part in the file
> name?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/20767#issuecomment-383579698>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIqGuDliwjnIcLB8pYPZytFQzt3dfks5trdpfgaJpZM4Tdo8r>
> .
>

> By file name, do you mean the environment.yaml? Because if so, I think that's the case.

Yes, that is what I meant. All environment files for travis are like `ci/travis-specifics.yml`, so JOB could just indicate 'specifics' (which would almost be identical to the current job tag, except for casing) to avoid duplication between JOB and ENV_FILE ?
Re-pushed with more informative job tags. Now they include things like indicating which one is linting, running coverage, etc.

![screen shot 2018-04-23 at 1 02 14 pm](https://user-images.githubusercontent.com/1312546/39144506-9796963c-46f6-11e8-8d0c-3a34bf73c4ff.png)

Easier to tell at a glance which ones are special.
lgtm

merge away
FYI, the `channel::package` support in environment.yaml was recently added to conda in https://github.com/conda/conda/pull/7178

I tried it out with conda-canary and things seemed to work fine.