CI: unify environment creation 
State: closed

closes #24498
closes #23923



Labels:
name: CI
description: Continuous Integration

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24632?src=pr&el=h1) Report
> Merging [#24632](https://codecov.io/gh/pandas-dev/pandas/pull/24632?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/cb31b2b0912c99e40b22a26e598e640548ea8f14?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24632/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24632?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24632      +/-   ##
==========================================
- Coverage   92.38%   92.37%   -0.01%     
==========================================
  Files         166      166              
  Lines       52395    52395              
==========================================
- Hits        48403    48402       -1     
- Misses       3992     3993       +1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.8% <ø> (ø)` | :arrow_up: |
| #single | `43% <ø> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24632?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/24632/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `88% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24632?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24632?src=pr&el=footer). Last update [cb31b2b...a8d22f3](https://codecov.io/gh/pandas-dev/pandas/pull/24632?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24632?src=pr&el=h1) Report
> Merging [#24632](https://codecov.io/gh/pandas-dev/pandas/pull/24632?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/de3a85c8404505ace0c0febc26b1fbe824f48034?src=pr&el=desc) will **decrease** coverage by `0.04%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24632/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24632?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24632      +/-   ##
==========================================
- Coverage   91.81%   91.77%   -0.05%     
==========================================
  Files         175      175              
  Lines       52580    52580              
==========================================
- Hits        48278    48255      -23     
- Misses       4302     4325      +23
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.33% <ø> (-0.04%)` | :arrow_down: |
| #single | `40.72% <ø> (-1.26%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24632?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/clipboard/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/24632/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NsaXBib2FyZC9fX2luaXRfXy5weQ==) | `39.21% <0%> (-17.65%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/24632/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `75% <0%> (-12.5%)` | :arrow_down: |
| [pandas/io/clipboard/clipboards.py](https://codecov.io/gh/pandas-dev/pandas/pull/24632/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NsaXBib2FyZC9jbGlwYm9hcmRzLnB5) | `18.51% <0%> (-12.35%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/24632/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.79% <0%> (-0.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24632?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24632?src=pr&el=footer). Last update [de3a85c...e400387](https://codecov.io/gh/pandas-dev/pandas/pull/24632?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

This sets up DB in both travis and azure. Have not tested DB jobs on Azure but maybe worth a try. 
yes remove the original files
and yes, please delete the replaced files in this same PR.
@datapythonista  - Azure is failing on conda update specifically - `conda update -q conda`. I was reading [here](https://github.com/conda/conda/issues/3869#issuecomment-384781590) that conda now suggests `conda update -n base conda`. I can change the script to reflect this. However, why are we even doing this? We are downloading and installing the latest miniconda build. Why would we need to update conda just after that. 

Another question is on line 32 of `setup_env.sh`. Why is the `rm -rf ` is needed in the current setup. I would think that the VM running the tests is spun fresh everytime we start the test.
Yes, please update the conda update statement to the new syntax. Anaconda can release a new conda version, but not create a new installable immediately, that's why it's common to update conda itself after installation. It's similar to when you install Linux and immediately run `dnf update` or equivalent. The iso is older than the versions of the latest packages.

I'm not sure about the `rm -rf`. I guess travis can reuse the containers or something. If the environment is clean for every build, it surely doesn't make sense. But please leave it for now, I'm sure it was added for a reason. 
 Pending items -

- DB works only on Travis
- Not sure where which test require xvfb. But I could not start it on Azure. I do not think this was enabled in Arure earlier as well. 

I am not inclined to merge this until *after* @TomAugspurger cuts the RC (haven't really looked yet). but the CI churn recently has caused too many issues.
I'll take a look at the caching once this is merged, thanks for the info, I don't know much about that part.

I think this not as likely to fail silently as the previous changes to the tests. But surely ok to wait.
@datapythonista / @jreback - any thoughts on this PR? 
this needs to wait until after 0.24.1 is out at a minimum
can you merge master and i'll loo again here
@jreback - green. 
@saurav2608 pls confirm for each build that we are running the same number of tests both before and after. This change could easy miss testing things and we would not easily know. (take the build on master right before you merged). check on both travis and azure; link the builds here.
@jreback : I compared this build to one just before. Details - 

PR|platform | No of Tests | Link 
--|---------|------|--------|
|#24632    |Azure | 336,141|https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=8496&view=ms.vss-test-web.build-test-results-tab
|#25209 |Azure |336,141|https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=8495&view=ms.vss-test-web.build-test-results-tab

For Travis the order was a bit different. But I compared the this build with the one just before. And compared 4 builds

PR>build>Job|platform | No of Tests | Link 
--------------|---------|-------------|--------|
#24632 > Build #52470 > Job #52470.1 | Travis | 43095 passed, 2814 skipped, 1008 xfailed, 20 xpassed, 58 warnings in 678.23 seconds | https://travis-ci.org/pandas-dev/pandas/jobs/494931336
#25362 > Build #52468 > Job #52468.1 | Travis | 43095 passed, 2814 skipped, 1008 xfailed, 20 xpassed, 58 warnings in 659.78 seconds | https://travis-ci.org/pandas-dev/pandas/jobs/494917125
#24632 > Build #52470 > Job #52470.2 | Travis | 45407 passed, 1325 skipped, 783 xfailed, 20 xpassed, 24 warnings in 1144.70 seconds  | https://travis-ci.org/pandas-dev/pandas/jobs/494931338
#25362 > Build #52468 > Job #52468.2 | Travis | 45407 passed, 1325 skipped, 783 xfailed, 20 xpassed, 24 warnings in 1093.64 seconds  | https://travis-ci.org/pandas-dev/pandas/jobs/494917126
#24632 > Build #52470 > Job #52470.3 | Travis | 45338 passed, 917 skipped, 1013 xfailed, 20 xpassed, 42 warnings in 880.59 seconds| https://travis-ci.org/pandas-dev/pandas/jobs/494931340
#25362 > Build #52468 > Job #52468.3 | Travis | 45338 passed, 917 skipped, 1013 xfailed, 20 xpassed, 42 warnings in 833.46 seconds |https://travis-ci.org/pandas-dev/pandas/jobs/494917127
#24632 > Build #52470 > Job #52470.4 | Travis | 45410 passed, 842 skipped, 1014 xfailed, 21 xpassed, 90 warnings in 1305.06 seconds | https://travis-ci.org/pandas-dev/pandas/jobs/494931341
#25362 > Build #52468 > Job #52468.4 | Travis | 45410 passed, 842 skipped, 1014 xfailed, 21 xpassed, 90 warnings in 1243.80 seconds   |https://travis-ci.org/pandas-dev/pandas/jobs/494917128
@saurav2608 ok this looks fine. I still am going to wait till we do 0.24.2, we will have to backport this as well I think. We have had CI brokeness in the past from even minor changes in the CI code.
Can you merge master on this?
What's the status of this PR? I constantly need to look up manually which version is in failing posix environments. If this is still taking longer, can we merge #25377 in the interim?
> What's the status of this PR? I constantly need to look up manually which version is in failing posix environments. If this is still taking longer, can we merge #25377 in the interim?
this needs a rebase and the PY2 and numpy bump merged first.
@saurav2608 if you wan to rebase this, i think we can do this now.
> @saurav2608 if you wan to rebase this, i think we can do this now.

@jreback - green
thanks @jreback and thanks @datapythonista for guiding this along.