CI/DOC: Building the documentation with azure-pipelines
State: closed

- [X] xref #22766, #26574
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Looks like the problem in #26591 was that the `HostName` input of the azure task that installs the key, is not the host name, but the `know_hosts` line for the host name where to push.

I set up azure-pipelines in my pandas branch, to be able to see the `master` builds and avoid surprises, and this branch is successfully pushing the docs to https://dev.pandas.io

@TomAugspurger since there were differences between the travis docs dependencies and `environment.yml` I think it makes more sense to change and use `environment.yml` here, so we can detect and fix any problem while we still have the travis docs, and then simply remove the travis docs dependencies with the build when we are happy with this.

@jorisvandenbossche the `ImportError` for `pyarrow` doesn't seem to be because `pyarrow` is not in `environment.yml`, it is. There is something weird with that, need to have a look.

Labels:
name: Docs
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26648?src=pr&el=h1) Report
> Merging [#26648](https://codecov.io/gh/pandas-dev/pandas/pull/26648?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/c07d71d13b21e0b6e22146f0f546f1f8e24a64b3?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26648/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26648?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26648      +/-   ##
==========================================
- Coverage   91.87%   91.87%   -0.01%     
==========================================
  Files         174      174              
  Lines       50692    50692              
==========================================
- Hits        46575    46571       -4     
- Misses       4117     4121       +4
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.41% <ø> (ø)` | :arrow_up: |
| #single | `41.78% <ø> (-0.09%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26648?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26648/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `78.94% <0%> (-10.53%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26648/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97% <0%> (-0.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26648?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26648?src=pr&el=footer). Last update [c07d71d...c959c2c](https://codecov.io/gh/pandas-dev/pandas/pull/26648?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26648?src=pr&el=h1) Report
> Merging [#26648](https://codecov.io/gh/pandas-dev/pandas/pull/26648?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/0f3e8e8bd2118d56c7febe5d6081c5058cb22be7?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26648/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26648?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26648      +/-   ##
==========================================
+ Coverage   91.71%   91.73%   +0.01%     
==========================================
  Files         178      178              
  Lines       50740    50774      +34     
==========================================
+ Hits        46538    46575      +37     
+ Misses       4202     4199       -3
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.32% <ø> (+0.02%)` | :arrow_up: |
| #single | `41.19% <ø> (-0.13%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26648?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26648/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `78.94% <0%> (-10.53%)` | :arrow_down: |
| [pandas/core/indexes/interval.py](https://codecov.io/gh/pandas-dev/pandas/pull/26648/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9pbnRlcnZhbC5weQ==) | `96.11% <0%> (-0.32%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26648/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.88% <0%> (-0.12%)` | :arrow_down: |
| [pandas/plotting/\_matplotlib/tools.py](https://codecov.io/gh/pandas-dev/pandas/pull/26648/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19tYXRwbG90bGliL3Rvb2xzLnB5) | `78.4% <0%> (ø)` | :arrow_up: |
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/26648/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `96.37% <0%> (ø)` | :arrow_up: |
| [pandas/plotting/\_matplotlib/style.py](https://codecov.io/gh/pandas-dev/pandas/pull/26648/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19tYXRwbG90bGliL3N0eWxlLnB5) | `85.96% <0%> (ø)` | :arrow_up: |
| [pandas/core/indexes/timedeltas.py](https://codecov.io/gh/pandas-dev/pandas/pull/26648/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy90aW1lZGVsdGFzLnB5) | `90.96% <0%> (ø)` | :arrow_up: |
| [pandas/plotting/\_matplotlib/timeseries.py](https://codecov.io/gh/pandas-dev/pandas/pull/26648/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19tYXRwbG90bGliL3RpbWVzZXJpZXMucHk=) | `66.32% <0%> (ø)` | :arrow_up: |
| [pandas/core/indexes/datetimelike.py](https://codecov.io/gh/pandas-dev/pandas/pull/26648/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZWxpa2UucHk=) | `98.15% <0%> (ø)` | :arrow_up: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/26648/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `93.64% <0%> (+0.02%)` | :arrow_up: |
| ... and [7 more](https://codecov.io/gh/pandas-dev/pandas/pull/26648/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26648?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26648?src=pr&el=footer). Last update [0f3e8e8...7c945fd](https://codecov.io/gh/pandas-dev/pandas/pull/26648?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

> @jorisvandenbossche the ImportError for pyarrow doesn't seem to be because pyarrow is not in environment.yml, it is. There is something weird with that, need to have a look.

Ah, yes, indeed. Looking at the error message now, it is indeed a problem with the installation, not that it is not installed. 
Possible reason is a mixture of defaults and conda-forge (eg arrow-cpp is from conda-forge, but pyarrow from defaults)
I think the problem is that `pyarrow` manages to install without a dependency or with something broken, but then it fails on import. We should probably append the original error message to the error we raise.

```
ImportError: /home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNK5arrow6Status8ToStringB5cxx11Ev
```

Any idea on what is missing?
I think it is related to the mentioned defaults/conda-forge mixing, but why conda is doing that, not sure ..
I see, not sure why `arrow-cpp` is installed from conda-forge with `environment.yml`. I'm in a windows machine right now, and I get the same, but `pyarrow` is working. Another difference is that with `environment.yml` is using Python 3.7 and not 3.6. But again, in my local set up, `pyarrow` still works with it.

I'll make azure build with `travis-36-doc.yaml` and see if that fixes the problem, otherwise I'm not sure what else we can do.
Looks like building with `ci/deps/travis-36-doc.yaml` doesn't cause the `pyarrow` problems in azure. Using that here, so we can merge this, and start to build the docs from azure.

I added the missing dependencies in `environment.yml` in #26657 and I opened #26656 for the pyarrow error.
It's very weird, but the CI was green, and eventually it became red because instead of referencing its own build, was referencing a build of a different PR (https://travis-ci.org/pandas-dev/pandas/jobs/541912722).

I rebased, hope this fix it, but mentioning here in case someone else sees the same, we should probably report the problem to github.
The clipboard is not yet working, see here how I recently fixed it on Travis (https://github.com/pandas-dev/pandas/pull/26103/files), but no idea if something similar can fix it on appveyor (might actually also be a good question if the tests for this are working).
Thanks @jorisvandenbossche I saw the warning, but thought it was one of the warnings that also happens in Travis.

I opened #26700 to address it separately. No change should be required here.
@jreback made the changes so the CI is still green even if the push of the documentation from azure fails. Tested both that the push works, and that if it fails the CI stays green.

Once merged, this will start updating dev.pandas.io with the master version of the docs.
@jorisvandenbossche can you have a look here? I think would be good to get this merged some time before we activate the fail on warning in the docs.
@jorisvandenbossche addressed your comments, CI is green
OK, let's try again! :)
thanks @jorisvandenbossche, CI is green, and docs where successfully built and pushed to http://dev.pandas.io/ :)