Repository: pandas-dev/pandas
Pull request: CI: Linting with azure instead of travis
closes #22844

Moving all the linting, other code checks and doctests to azure. 

Problem statement:
BLD: Use Azure pipelines for linting
At the moment we have the script `ci/lint.sh` for validating PEP-8 and other linting. The script has several validations in the form of:
```
   echo "Linting *.py"
    flake8 pandas --filename=*.py --exclude pandas/_libs/src --ignore=C406,C408,C409,E402,E731,E741,W503
    if [ $? -ne "0" ]; then
        RET=1
    fi
```

If I'm not wrong, moving all those to the azure-pipelines configuration would have two advantages.

First, the code would be simpler, as all the bash stuff would be automatically managed, and we'ld simply have:
```
  - script: flake8 pandas --filename=*.py --exclude pandas/_libs/src --ignore=C406,C408,C409,E402,E731,E741,W503
displayName: 'Linting *.py'
```

Second is that when one of those fails, it'll be faster to see which is the reason for the fail in the azure pipeline dashboard, instead of having to go into the details of the travis log.

Does this make sense? @jreback @TomAugspurger @chrisrpatterson @jeremyepling

Hints from issue discussion:
> First, the code would be simpler, as all the bash stuff would be automatically managed

I think we'll still need something similar to what's currently in `lint.sh`, since we have custom checks that aren't part of flake8.

I agree we should do something to make seeing lint failures easier for users. I'm not sure whether this needs to be done on azure or not. I haven't played with more complex pipelines or reporting, beyond the basic "here's a file of junit XML results".


I was having a look, and I think for the ones that are a single command (`flake8`, `cpplint`, `grep`...) should be just as the example in the description.

If I'm not wrong, the only other case that we've got are loops. I guess it's possible to have the whole loop as the script, but that wouldn't show directly which file failed (it'd probably be an improvement, even if to know the exact file we need to check the log). But I think it'd be even better, if azure provides a way to create a separate item for the values of the loop. Not sure if that's doable or how complex would the configuration. This is a case:
```
    echo "Linting *.c and *.h"
    for path in '*.h' 'parser' 'ujson'
    do
        echo "linting -> pandas/_libs/src/$path"
        cpplint --quiet --extensions=c,h --headers=h --filter=-readability/casting,-runtime/int,-build/include_subdir --recursive pandas/_libs/src/$path
        if [ $? -ne "0" ]; then
            RET=1
        fi
    done
```
@datapythonista we have a model for highlighting issues in the log and in the summary view that also flows back to checks.

You can echo out 
##vso[task.logissue type=error;sourcepath=consoleapp/main.cs;linenumber=1;columnnumber=1;code=100;]this is an error

And that will get a treatment on the summary view that will include a link to the sourcefile and line number where the issues are found.  For more details see https://github.com/Microsoft/vsts-tasks/blob/master/docs/authoring/commands.md. 
Thanks for the info @chrisrpatterson. That would be really nice to have. But I don't see how can I get this with flake8, which is generating the messages. The output we've got from the command is something like this:
```
stat_ops.py:110:5: F811 redefinition of unused 'setup' from line 4
strings.py:95:9: E731 do not assign a lambda expression, use a def
timeseries.py:1:1: F401 'warnings' imported but unused
```
Is there a "standards" way in Azure to change the format? We could do a script that changes from one format to another, and call `flake8 . | azure_log_flake8_formatter`, but seems suboptimal that every project has to implement the `azure_log_flake8_formatter`.

@zooba suggested to start the lines with `##[error]`, which has the same problem (except that it's easier to implement with awk: `flake8 . | awk '{print "##[error] " $0}'`, but that looses the exit status and would make the build succeed), so I'm not sure if I'm missing something here.


ok, in a first check to flake8 parameters I didn't see the format could be specified, but I double checked, and everything makes sense now: `flake8 --format="##vso[task.logissue type=error;sourcepath=%(path)s;linenumber=%(row)s;columnnumber=%(col)s;code=%(code)s;]%(text)s"`

Will give it a try.
`
Made some progress on this, and managed to make flake8 generate the correct output:

https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=437&view=logs

It really improves the clarity of the errors (and even much more compared to our current solution, in the travis log). But still some issues:
- cpplint doesn't seem to allow custom output (it has a `--output` parameter that can be `emacs`, `vs7`, `eclipse`... but not a custom one)
- The number of errors reported by azure is wrong. We've got like 50 errors in the first failing step if we enter the log, but in the preview there are 11, and it says we've got just 11. Weird
- The link is not created. I see there is a `##vso[task.setendpoint]value` option in https://github.com/Microsoft/vsts-tasks/blob/master/docs/authoring/commands.md, but it's unclear how this needs to be used.

We probably don't want to change the `flake8` format in `setup.cfg` (would change it when `flake8` is run locally too). But in any case, doesn't seem to work, as the `%(path)s` are rendered by `ConfigParser`, and it fails to install flake8 in azure.
We have been having some discussions about other ways to make this sort of parsing a bit more generic but nothing has come to us yet.  However, similar to test results there are probably a relatively small number of regex based formats we could build in support for recognizing.  The other option might be to have a generic regex you could specify as part of the job or even the step that would be used for parsing out issues.

Commit message: Adding conda environment to run scripts tests, and changing verbosity of doctests

Patch to review:
diff --git a/.travis.yml b/.travis.yml
index 6d31adcbf8a43..0140688b8e492 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -14,13 +14,6 @@ cache:
   - $HOME/.cache # cython cache
   - $HOME/.ccache # compiler cache
 
-env:
-  global:
-    # create a github personal access token
-    # cd pandas-dev/pandas
-    # travis encrypt 'PANDAS_GH_TOKEN=personal_access_token' -r pandas-dev/pandas
-    - secure: "EkWLZhbrp/mXJOx38CHjs7BnjXafsqHtwxPQrqWy457VDFWhIY1DMnIR/lOWG+a20Qv52sCsFtiZEmMfUjf0pLGXOqurdxbYBGJ7/ikFLk9yV2rDwiArUlVM9bWFnFxHvdz9zewBH55WurrY4ShZWyV+x2dWjjceWG5VpWeI6sA="
-
 git:
     # for cloning
     depth: 1500
@@ -52,7 +45,7 @@ matrix:
           - python-gtk2
     - dist: trusty
       env:
-        - JOB="3.6, lint, coverage" ENV_FILE="ci/deps/travis-36.yaml" TEST_ARGS="--skip-slow --skip-network" PANDAS_TESTING_MODE="deprecate" COVERAGE=true LINT=true
+        - JOB="3.6, coverage" ENV_FILE="ci/deps/travis-36.yaml" TEST_ARGS="--skip-slow --skip-network" PANDAS_TESTING_MODE="deprecate" COVERAGE=true
     - dist: trusty
       env:
         - JOB="3.7, NumPy dev" ENV_FILE="ci/deps/travis-37-numpydev.yaml" TEST_ARGS="--skip-slow --skip-network -W error" PANDAS_TESTING_MODE="deprecate"
@@ -66,17 +59,10 @@ matrix:
       env:
         - JOB="3.6, slow" ENV_FILE="ci/deps/travis-36-slow.yaml" SLOW=true
 
-    # In allow_failures
-    - dist: trusty
-      env:
-        - JOB="3.6, doc" ENV_FILE="ci/deps/travis-36-doc.yaml" DOC=true
     allow_failures:
       - dist: trusty
         env:
           - JOB="3.6, slow" ENV_FILE="ci/deps/travis-36-slow.yaml" SLOW=true
-      - dist: trusty
-        env:
-          - JOB="3.6, doc" ENV_FILE="ci/deps/travis-36-doc.yaml" DOC=true
 
 before_install:
   - echo "before_install"
@@ -100,15 +86,14 @@ install:
 
 before_script:
   - ci/install_db_travis.sh
-  - export DISPLAY=":99.0"
+  # XXX checking if this is still required now that doctests are in azure
+  # - export DISPLAY=":99.0"
   - ci/before_script_travis.sh
 
 script:
   - echo "script start"
-  - ci/run_build_docs.sh
   - ci/script_single.sh
   - ci/script_multi.sh
-  - ci/code_checks.sh
 
 after_success:
   -  ci/upload_coverage.sh
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
index 373c22fdf8e62..eef05592fef14 100644
--- a/azure-pipelines.yml
+++ b/azure-pipelines.yml
@@ -23,3 +23,124 @@ jobs:
   parameters:
     name: WindowsPy27
     vmImage: vs2017-win2016
+
+- job: 'Checks_and_doc'
+  pool:
+    vmImage: ubuntu-16.04
+  steps:
+  - task: UsePythonVersion@0
+    inputs:
+      versionSpec: '3.7'
+      architecture: 'x64'
+
+  - script: |
+      # XXX not sure why setting $PATH here does not work
+      # echo '##vso[task.setvariable variable=PATH]$HOME/miniconda3/bin:$PATH'
+      echo '##vso[task.setvariable variable=CONDA_ENV]pandas-dev'
+      echo '##vso[task.setvariable variable=ENV_FILE]environment.yml'
+      echo '##vso[task.setvariable variable=AZURE]true'
+    displayName: 'Setting environment variables'
+
+  # Do not require a conda environment
+  - script: |
+      export PATH=$HOME/miniconda3/bin:$PATH
+      source activate pandas-dev
+      ci/code_checks.sh patterns
+    displayName: 'Looking for unwanted patterns'
+    condition: true
+
+  - script: |
+      export PATH=$HOME/miniconda3/bin:$PATH
+      sudo apt-get install -y libc6-dev-i386
+      ci/incremental/install_miniconda.sh
+      ci/incremental/setup_conda_environment.sh
+    displayName: 'Set up environment'
+
+  # Do not require pandas
+  - script: |
+      export PATH=$HOME/miniconda3/bin:$PATH
+      source activate pandas-dev
+      ci/code_checks.sh lint
+    displayName: 'Linting'
+    condition: true
+
+  - script: |
+      export PATH=$HOME/miniconda3/bin:$PATH
+      source activate pandas-dev
+      ci/code_checks.sh dependencies
+    displayName: 'Dependencies consistency'
+    condition: true
+
+  - script: |
+      export PATH=$HOME/miniconda3/bin:$PATH
+      ci/incremental/build.sh
+    displayName: 'Build'
+    condition: true
+
+  # Require pandas
+  - script: |
+      export PATH=$HOME/miniconda3/bin:$PATH
+      source activate pandas-dev
+      ci/code_checks.sh code
+    displayName: 'Checks on imported code'
+    condition: true
+
+  - script: |
+      export PATH=$HOME/miniconda3/bin:$PATH
+      source activate pandas-dev
+      ci/code_checks.sh doctests
+    displayName: 'Running doctests'
+    condition: true
+
+  - script: |
+      export PATH=$HOME/miniconda3/bin:$PATH
+      source activate pandas-dev
+      ci/code_checks.sh docstrings
+    displayName: 'Docstring validation'
+    condition: true
+
+  - script: |
+      export PATH=$HOME/miniconda3/bin:$PATH
+      source activate pandas-dev
+      pytest --capture=no --strict scripts
+    displayName: 'Testing docstring validaton script'
+    condition: true
+
+  - script: |
+      export PATH=$HOME/miniconda3/bin:$PATH
+      source activate pandas-dev
+      doc/make.py html
+    displayName: 'Building docs'
+    condition: true
+
+  - script: |
+      az extension add --name storage-preview
+      az storage blob upload-batch -s $SOURCE_PATH -d '$web'
+    displayName: 'Publishing docs (Azure storage)'
+    condition: true
+    env:
+      SOURCE_PATH: $(Build.SourcesDirectory)/doc/build/html/
+      AZURE_STORAGE_CONNECTION_STRING: "DefaultEndpointsProtocol=https;AccountName=$(AZURE_STORAGE_ACCOUNT);AccountKey=$(AZURE_STORAGE_ACCESS_KEY)"
+
+  - script: |
+      if [ "$(Build.SourceBranch)" == "refs/heads/master" ]; then
+          export NAME="master"
+      else
+          export NAME=$(System.PullRequest.PullRequestId)
+      fi
+      export PR_DIR=$REPO_DIR/pr/$NAME
+      git clone https://$(GITHUB_DOC_TOKEN)@github.com/pandas-dev/pandas-docs-travis.git $REPO_DIR
+      mkdir -p $PR_DIR
+      rm -rf $PR_DIR/*
+      cp -r doc/build/html/* $PR_DIR/
+      cp LICENSE $PR_DIR/
+      cd $REPO_DIR
+      git add --all .
+      git config user.email "pandas-docs-bot@localhost"
+      git config user.name "pandas-docs-bot"
+      git commit -m "Added pandas documentation for $NAME"
+      git push
+    displayName: 'Publishing docs (GitHub pages)'
+    condition: true
+    env:
+      REPO_DIR: $(Build.ArtifactStagingDirectory)/pandas-docs-travis
diff --git a/ci/build_docs.sh b/ci/build_docs.sh
deleted file mode 100755
index 33340a1c038dc..0000000000000
--- a/ci/build_docs.sh
+++ /dev/null
@@ -1,52 +0,0 @@
-#!/bin/bash
-
-if [ "${TRAVIS_OS_NAME}" != "linux" ]; then
-   echo "not doing build_docs on non-linux"
-   exit 0
-fi
-
-cd "$TRAVIS_BUILD_DIR"/doc
-echo "inside $0"
-
-if [ "$DOC" ]; then
-
-    echo "Will build docs"
-
-    source activate pandas
-
-    echo ###############################
-    echo # Log file for the doc build  #
-    echo ###############################
-
-    echo ./make.py
-    ./make.py
-
-    echo ########################
-    echo # Create and send docs #
-    echo ########################
-
-    cd build/html
-    git config --global user.email "pandas-docs-bot@localhost.foo"
-    git config --global user.name "pandas-docs-bot"
-
-    # create the repo
-    git init
-
-    touch README
-    git add README
-    git commit -m "Initial commit" --allow-empty
-    git branch gh-pages
-    git checkout gh-pages
-    touch .nojekyll
-    git add --all .
-    git commit -m "Version" --allow-empty
-
-    git remote remove origin
-    git remote add origin "https://${PANDAS_GH_TOKEN}@github.com/pandas-dev/pandas-docs-travis.git"
-    git fetch origin
-    git remote -v
-
-    git push origin gh-pages -f
-fi
-
-exit 0
diff --git a/ci/code_checks.sh b/ci/code_checks.sh
index f0772f72d63d4..ef5095417d6f8 100755
--- a/ci/code_checks.sh
+++ b/ci/code_checks.sh
@@ -5,26 +5,48 @@
 # This script is intended for both the CI and to check locally that code standards are
 # respected. We are currently linting (PEP-8 and similar), looking for patterns of
 # common mistakes (sphinx directives with missing blank lines, old style classes,
-# unwanted imports...), and we also run doctests here (currently some files only).
-# In the future we may want to add the validation of docstrings and other checks here.
+# unwanted imports...), we run doctests here (currently some files only), and we
+# validate formatting error in docstrings.
 #
 # Usage:
 #   $ ./ci/code_checks.sh               # run all checks
 #   $ ./ci/code_checks.sh lint          # run linting only
 #   $ ./ci/code_checks.sh patterns      # check for patterns that should not exist
+#   $ ./ci/code_checks.sh code          # checks on imported code
 #   $ ./ci/code_checks.sh doctests      # run doctests
+#   $ ./ci/code_checks.sh docstrings    # validate docstring errors
 #   $ ./ci/code_checks.sh dependencies  # check that dependencies are consistent
 
-echo "inside $0"
-[[ $LINT ]] || { echo "NOT Linting. To lint use: LINT=true $0 $1"; exit 0; }
-[[ -z "$1" || "$1" == "lint" || "$1" == "patterns" || "$1" == "doctests" || "$1" == "dependencies"  ]] \
-    || { echo "Unknown command $1. Usage: $0 [lint|patterns|doctests|dependencies]"; exit 9999; }
+[[ -z "$1" || "$1" == "lint" || "$1" == "patterns" || "$1" == "code" || "$1" == "doctests" || "$1" == "docstrings" || "$1" == "dependencies" ]] || \
+    { echo "Unknown command $1. Usage: $0 [lint|patterns|code|doctests|docstrings|dependencies]"; exit 9999; }
 
-source activate pandas
 BASE_DIR="$(dirname $0)/.."
 RET=0
 CHECK=$1
 
+function invgrep {
+    # grep with inverse exist status and formatting for azure-pipelines
+    #
+    # This function works exactly as grep, but with opposite exit status:
+    # - 0 (success) when no patterns are found
+    # - 1 (fail) when the patterns are found
+    #
+    # This is useful for the CI, as we want to fail if one of the patterns
+    # that we want to avoid is found by grep.
+    if [[ "$AZURE" == "true" ]]; then
+        set -o pipefail
+        grep -n "$@" | awk -F ":" '{print "##vso[task.logissue type=error;sourcepath=" $1 ";linenumber=" $2 ";] Found unwanted pattern: " $3}'
+    else
+        grep "$@"
+    fi
+    return $((! $?))
+}
+
+if [[ "$AZURE" == "true" ]]; then
+    FLAKE8_FORMAT="##vso[task.logissue type=error;sourcepath=%(path)s;linenumber=%(row)s;columnnumber=%(col)s;code=%(code)s;]%(text)s"
+else
+    FLAKE8_FORMAT="default"
+fi
 
 ### LINTING ###
 if [[ -z "$CHECK" || "$CHECK" == "lint" ]]; then
@@ -36,22 +58,22 @@ if [[ -z "$CHECK" || "$CHECK" == "lint" ]]; then
 
     # pandas/_libs/src is C code, so no need to search there.
     MSG='Linting .py code' ; echo $MSG
-    flake8 .
+    flake8 --format="$FLAKE8_FORMAT" .
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
     MSG='Linting .pyx code' ; echo $MSG
-    flake8 pandas --filename=*.pyx --select=E501,E302,E203,E111,E114,E221,E303,E128,E231,E126,E265,E305,E301,E127,E261,E271,E129,W291,E222,E241,E123,F403,C400,C401,C402,C403,C404,C405,C406,C407,C408,C409,C410,C411
+    flake8 --format="$FLAKE8_FORMAT" pandas --filename=*.pyx --select=E501,E302,E203,E111,E114,E221,E303,E128,E231,E126,E265,E305,E301,E127,E261,E271,E129,W291,E222,E241,E123,F403,C400,C401,C402,C403,C404,C405,C406,C407,C408,C409,C410,C411
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
     MSG='Linting .pxd and .pxi.in' ; echo $MSG
-    flake8 pandas/_libs --filename=*.pxi.in,*.pxd --select=E501,E302,E203,E111,E114,E221,E303,E231,E126,F403
+    flake8 --format="$FLAKE8_FORMAT" pandas/_libs --filename=*.pxi.in,*.pxd --select=E501,E302,E203,E111,E114,E221,E303,E231,E126,F403
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
     echo "flake8-rst --version"
     flake8-rst --version
 
     MSG='Linting code-blocks in .rst documentation' ; echo $MSG
-    flake8-rst doc/source --filename=*.rst
+    flake8-rst doc/source --filename=*.rst --format="$FLAKE8_FORMAT"
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
     # Check that cython casting is of the form `<type>obj` as opposed to `<type> obj`;
@@ -59,7 +81,7 @@ if [[ -z "$CHECK" || "$CHECK" == "lint" ]]; then
     # Note: this grep pattern is (intended to be) equivalent to the python
     # regex r'(?<![ ->])> '
     MSG='Linting .pyx code for spacing conventions in casting' ; echo $MSG
-    ! grep -r -E --include '*.pyx' --include '*.pxi.in' '[a-zA-Z0-9*]> ' pandas/_libs
+    invgrep -r -E --include '*.pyx' --include '*.pxi.in' '[a-zA-Z0-9*]> ' pandas/_libs
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
     # readability/casting: Warnings about C casting instead of C++ casting
@@ -89,43 +111,48 @@ if [[ -z "$CHECK" || "$CHECK" == "patterns" ]]; then
 
     # Check for imports from pandas.core.common instead of `import pandas.core.common as com`
     MSG='Check for non-standard imports' ; echo $MSG
-    ! grep -R --include="*.py*" -E "from pandas.core.common import " pandas
+    invgrep -R --include="*.py*" -E "from pandas.core.common import " pandas
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
     MSG='Check for pytest warns' ; echo $MSG
-    ! grep -r -E --include '*.py' 'pytest\.warns' pandas/tests/
+    invgrep -r -E --include '*.py' 'pytest\.warns' pandas/tests/
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
     # Check for the following code in testing: `np.testing` and `np.array_equal`
     MSG='Check for invalid testing' ; echo $MSG
-    ! grep -r -E --include '*.py' --exclude testing.py '(numpy|np)(\.testing|\.array_equal)' pandas/tests/
+    invgrep -r -E --include '*.py' --exclude testing.py '(numpy|np)(\.testing|\.array_equal)' pandas/tests/
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
     # Check for the following code in the extension array base tests: `tm.assert_frame_equal` and `tm.assert_series_equal`
     MSG='Check for invalid EA testing' ; echo $MSG
-    ! grep -r -E --include '*.py' --exclude base.py 'tm.assert_(series|frame)_equal' pandas/tests/extension/base
+    invgrep -r -E --include '*.py' --exclude base.py 'tm.assert_(series|frame)_equal' pandas/tests/extension/base
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
     MSG='Check for deprecated messages without sphinx directive' ; echo $MSG
-    ! grep -R --include="*.py" --include="*.pyx" -E "(DEPRECATED|DEPRECATE|Deprecated)(:|,|\.)" pandas
+    invgrep -R --include="*.py" --include="*.pyx" -E "(DEPRECATED|DEPRECATE|Deprecated)(:|,|\.)" pandas
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
     MSG='Check for old-style classes' ; echo $MSG
-    ! grep -R --include="*.py" -E "class\s\S*[^)]:" pandas scripts
+    invgrep -R --include="*.py" -E "class\s\S*[^)]:" pandas scripts
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
     MSG='Check for backticks incorrectly rendering because of missing spaces' ; echo $MSG
-    ! grep -R --include="*.rst" -E "[a-zA-Z0-9]\`\`?[a-zA-Z0-9]" doc/source/
+    invgrep -R --include="*.rst" -E "[a-zA-Z0-9]\`\`?[a-zA-Z0-9]" doc/source/
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
     MSG='Check for incorrect sphinx directives' ; echo $MSG
-    ! grep -R --include="*.py" --include="*.pyx" --include="*.rst" -E "\.\. (autosummary|contents|currentmodule|deprecated|function|image|important|include|ipython|literalinclude|math|module|note|raw|seealso|toctree|versionadded|versionchanged|warning):[^:]" ./pandas ./doc/source
+    invgrep -R --include="*.py" --include="*.pyx" --include="*.rst" -E "\.\. (autosummary|contents|currentmodule|deprecated|function|image|important|include|ipython|literalinclude|math|module|note|raw|seealso|toctree|versionadded|versionchanged|warning):[^:]" ./pandas ./doc/source
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
     MSG='Check that the deprecated `assert_raises_regex` is not used (`pytest.raises(match=pattern)` should be used instead)' ; echo $MSG
     ! grep -R --exclude=*.pyc --exclude=testing.py --exclude=test_testing.py assert_raises_regex pandas
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
+fi
+
+### CODE ###
+if [[ -z "$CHECK" || "$CHECK" == "code" ]]; then
+
     MSG='Check for modules that pandas should not import' ; echo $MSG
     python -c "
 import sys
@@ -136,7 +163,7 @@ blacklist = {'bs4', 'gcsfs', 'html5lib', 'ipython', 'jinja2' 'hypothesis',
              'tables', 'xlrd', 'xlsxwriter', 'xlwt'}
 mods = blacklist & set(m.split('.')[0] for m in sys.modules)
 if mods:
-    sys.stderr.write('pandas should not import: {}\n'.format(', '.join(mods)))
+    sys.stderr.write('err: pandas should not import: {}\n'.format(', '.join(mods)))
     sys.exit(len(mods))
     "
     RET=$(($RET + $?)) ; echo $MSG "DONE"
@@ -158,7 +185,7 @@ if [[ -z "$CHECK" || "$CHECK" == "doctests" ]]; then
 
     MSG='Doctests generic.py' ; echo $MSG
     pytest -q --doctest-modules pandas/core/generic.py \
-        -k"-_set_axis_name -_xs -describe -droplevel -groupby -interpolate -pct_change -pipe -reindex -reindex_axis -to_json -transpose -values -xs"
+        -k"-_set_axis_name -_xs -describe -droplevel -groupby -interpolate -pct_change -pipe -reindex -reindex_axis -to_json -transpose -values -xs -to_clipboard"
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
     MSG='Doctests top-level reshaping functions' ; echo $MSG
@@ -179,11 +206,22 @@ if [[ -z "$CHECK" || "$CHECK" == "doctests" ]]; then
 
 fi
 
+### DOCSTRINGS ###
+if [[ -z "$CHECK" || "$CHECK" == "docstrings" ]]; then
+
+    MSG='Validate docstrings (GL06, SS04, PR03, PR05, EX04)' ; echo $MSG
+    $BASE_DIR/scripts/validate_docstrings.py --format=azure --errors=GL06,SS04,PR03,PR05,EX04
+    RET=$(($RET + $?)) ; echo $MSG "DONE"
+
+fi
+
 ### DEPENDENCIES ###
 if [[ -z "$CHECK" || "$CHECK" == "dependencies" ]]; then
+
     MSG='Check that requirements-dev.txt has been generated from environment.yml' ; echo $MSG
-    $BASE_DIR/scripts/generate_pip_deps_from_conda.py --compare
+    $BASE_DIR/scripts/generate_pip_deps_from_conda.py --compare --azure
     RET=$(($RET + $?)) ; echo $MSG "DONE"
+
 fi
 
 exit $RET
diff --git a/ci/deps/travis-36-doc.yaml b/ci/deps/travis-36-doc.yaml
deleted file mode 100644
index fb54c784d6fac..0000000000000
--- a/ci/deps/travis-36-doc.yaml
+++ /dev/null
@@ -1,44 +0,0 @@
-name: pandas
-channels:
-  - defaults
-  - conda-forge
-dependencies:
-  - beautifulsoup4
-  - bottleneck
-  - cython>=0.28.2
-  - fastparquet
-  - gitpython
-  - html5lib
-  - hypothesis>=3.58.0
-  - ipykernel
-  - ipython
-  - ipywidgets
-  - lxml
-  - matplotlib
-  - nbconvert
-  - nbformat
-  - nbsphinx
-  - notebook
-  - numexpr
-  - numpy=1.13*
-  - openpyxl
-  - pandoc
-  - pyarrow
-  - pyqt
-  - pytables
-  - python-dateutil
-  - python-snappy
-  - python=3.6*
-  - pytz
-  - scipy
-  - seaborn
-  - sphinx
-  - sqlalchemy
-  - statsmodels
-  - xarray
-  - xlrd
-  - xlsxwriter
-  - xlwt
-  # universal
-  - pytest
-  - pytest-xdist
diff --git a/ci/deps/travis-36.yaml b/ci/deps/travis-36.yaml
index 1880fa2501581..f67fa1d5713d6 100644
--- a/ci/deps/travis-36.yaml
+++ b/ci/deps/travis-36.yaml
@@ -7,16 +7,9 @@ dependencies:
   - cython>=0.28.2
   - dask
   - fastparquet
-  - flake8>=3.5
-  - flake8-comprehensions
-  - flake8-rst=0.4.2
   - gcsfs
   - geopandas
   - html5lib
-  - ipython
-  - isort
-  - jinja2
-  - lxml
   - matplotlib
   - nomkl
   - numexpr
@@ -32,7 +25,6 @@ dependencies:
   - s3fs
   - scikit-learn
   - scipy
-  - seaborn
   - sqlalchemy
   - statsmodels
   - xarray
@@ -48,6 +40,5 @@ dependencies:
   - pip:
     - brotlipy
     - coverage
-    - cpplint
     - pandas-datareader
     - python-dateutil
diff --git a/ci/run_build_docs.sh b/ci/run_build_docs.sh
deleted file mode 100755
index 2909b9619552e..0000000000000
--- a/ci/run_build_docs.sh
+++ /dev/null
@@ -1,10 +0,0 @@
-#!/bin/bash
-
-echo "inside $0"
-
-"$TRAVIS_BUILD_DIR"/ci/build_docs.sh 2>&1
-
-# wait until subprocesses finish (build_docs.sh)
-wait
-
-exit 0
diff --git a/ci/script_multi.sh b/ci/script_multi.sh
index e56d5da7232b2..2fbda3cdd2792 100755
--- a/ci/script_multi.sh
+++ b/ci/script_multi.sh
@@ -23,10 +23,7 @@ fi
 export PYTHONHASHSEED=$(python -c 'import random; print(random.randint(1, 4294967295))')
 echo PYTHONHASHSEED=$PYTHONHASHSEED
 
-if [ "$DOC" ]; then
-    echo "We are not running pytest as this is a doc-build"
-
-elif [ "$COVERAGE" ]; then
+if [ "$COVERAGE" ]; then
     echo pytest -s -n 2 -m "not single" --durations=10 --cov=pandas --cov-report xml:/tmp/cov-multiple.xml --junitxml=test-data-multiple.xml --strict $TEST_ARGS pandas
     pytest      -s -n 2 -m "not single" --durations=10 --cov=pandas --cov-report xml:/tmp/cov-multiple.xml --junitxml=test-data-multiple.xml --strict $TEST_ARGS pandas
 
diff --git a/ci/script_single.sh b/ci/script_single.sh
index ea0d48bc2da8a..fff019143bd73 100755
--- a/ci/script_single.sh
+++ b/ci/script_single.sh
@@ -22,14 +22,9 @@ if echo "$TEST_ARGS" | grep -e --skip-network -q; then
     export http_proxy=http://1.2.3.4 https_proxy=http://1.2.3.4;
 fi
 
-if [ "$DOC" ]; then
-    echo "We are not running pytest as this is a doc-build"
-
-elif [ "$COVERAGE" ]; then
+if [ "$COVERAGE" ]; then
     echo pytest -s -m "single" --durations=10 --strict --cov=pandas --cov-report xml:/tmp/cov-single.xml --junitxml=test-data-single.xml $TEST_ARGS pandas
     pytest      -s -m "single" --durations=10 --strict --cov=pandas --cov-report xml:/tmp/cov-single.xml --junitxml=test-data-single.xml $TEST_ARGS pandas
-    echo pytest -s --strict scripts
-    pytest      -s --strict scripts
 else
     echo pytest -m "single" --durations=10 --junitxml=test-data-single.xml --strict $TEST_ARGS pandas
     pytest      -m "single" --durations=10 --junitxml=test-data-single.xml --strict $TEST_ARGS pandas
diff --git a/environment.yml b/environment.yml
index fc35f1290f1b1..918da32920396 100644
--- a/environment.yml
+++ b/environment.yml
@@ -52,3 +52,5 @@ dependencies:
   - xlrd
   - xlsxwriter
   - xlwt
+  - pip:
+    - cpplint
diff --git a/requirements-dev.txt b/requirements-dev.txt
index d01a21ac5fed5..ac48ef620e4a9 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -40,4 +40,5 @@ statsmodels
 xarray
 xlrd
 xlsxwriter
-xlwt
\ No newline at end of file
+xlwt
+cpplint
\ No newline at end of file
diff --git a/scripts/generate_pip_deps_from_conda.py b/scripts/generate_pip_deps_from_conda.py
index 1f79b23a259dc..7b6eb1f9a32b5 100755
--- a/scripts/generate_pip_deps_from_conda.py
+++ b/scripts/generate_pip_deps_from_conda.py
@@ -75,7 +75,18 @@ def main(conda_fname, pip_fname, compare=False):
     with open(conda_fname) as conda_fd:
         deps = yaml.safe_load(conda_fd)['dependencies']
 
-    pip_content = '\n'.join(filter(None, map(conda_package_to_pip, deps)))
+    pip_deps = []
+    for dep in deps:
+        if isinstance(dep, str):
+            conda_dep = conda_package_to_pip(dep)
+            if conda_dep:
+                pip_deps.append(conda_dep)
+        elif isinstance(dep, dict) and len(dep) == 1 and 'pip' in dep:
+            pip_deps += dep['pip']
+        else:
+            raise ValueError('Unexpected dependency {}'.format(dep))
+
+    pip_content = '\n'.join(pip_deps)
 
     if compare:
         with open(pip_fname) as pip_fd:
@@ -92,6 +103,9 @@ def main(conda_fname, pip_fname, compare=False):
     argparser.add_argument('--compare',
                            action='store_true',
                            help='compare whether the two files are equivalent')
+    argparser.add_argument('--azure',
+                           action='store_true',
+                           help='show the output in azure-pipelines format')
     args = argparser.parse_args()
 
     repo_path = os.path.dirname(os.path.abspath(os.path.dirname(__file__)))
@@ -99,7 +113,10 @@ def main(conda_fname, pip_fname, compare=False):
                os.path.join(repo_path, 'requirements-dev.txt'),
                compare=args.compare)
     if res:
-        sys.stderr.write('`requirements-dev.txt` has to be generated with '
-                         '`{}` after `environment.yml` is modified.\n'.format(
-                             sys.argv[0]))
+        msg = ('`requirements-dev.txt` has to be generated with `{}` after '
+               '`environment.yml` is modified.\n'.format(sys.argv[0]))
+        if args.azure:
+            msg = ('##vso[task.logissue type=error;'
+                   'sourcepath=requirements-dev.txt]{}'.format(msg))
+        sys.stderr.write(msg)
     sys.exit(res)


Code review comments:
diff_hunk: @@ -14,13 +14,6 @@ cache:
   - $HOME/.cache # cython cache
   - $HOME/.ccache # compiler cache
 
-env:
-  global:
-    # create a github personal access token
-    # cd pandas-dev/pandas
path: .travis.yml
text: @user1:
i would leave the tracis doc build u til we are sure that the azure is actually working