Repository: scikit-learn/scikit-learn
Pull request: [MRG] Makes linting a dependency for most azure jobs
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes https://github.com/scikit-learn/scikit-learn/issues/15109


#### What does this implement/fix? Explain your changes.
Makes the linter a dependency of most jobs. The `pylatest_conda_mkl` job will run regardless of the linting outcome.


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->


Problem statement:
Use linting to for a requirement for some jobs on azure
Now that there are 8 jobs on azure, may we use linting as a dependency on some of these jobs to save resources? For example the following will always run:

1. `Linux32 py35_ubuntu_atlas_32bit`
2. `Linux pylatest_pip_openblas_pandas`
3. `macOS pylatest_conda_mkl`
4. `Windows py37_conda_mkl`

The rest will run linting first, and only when lint passes, the rest of the tests will run.

I hope this proposal is a good balance of lowering our carbon footprint and also have the benefits of having tests always run.

Hints from issue discussion:
+1

We may even just run one of them? Linting is very easy to check/fix locally
I'm okay with this

>

Commit message: STY Fixed

Patch to review:
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
index 9eca835865868..b614046317a10 100644
--- a/azure-pipelines.yml
+++ b/azure-pipelines.yml
@@ -1,9 +1,49 @@
 # Adapted from https://github.com/pandas-dev/pandas/blob/master/azure-pipelines.yml
 jobs:
+- job: linting
+  displayName: Linting
+  pool:
+    vmImage: ubuntu-16.04
+  steps:
+    - bash: echo "##vso[task.prependpath]$CONDA/bin"
+      displayName: Add conda to PATH
+    - bash: sudo chown -R $USER $CONDA
+      displayName: Take ownership of conda installation
+    - bash: conda create --name flake8_env --yes flake8
+      displayName: Install flake8
+    - bash: |
+        source activate flake8_env
+        ./build_tools/circle/flake8_diff.sh
+      displayName: Run flake8
+    - bash: |
+        source activate flake8_env
+        ./build_tools/circle/check_deprecated_properties.sh
+      displayName: Run deprecation properties checks
+
+
+# Will run all the time regardless of linting outcome.
+- template: build_tools/azure/posix.yml
+  parameters:
+    name: Linux_Always_Runs
+    vmImage: ubuntu-16.04
+    matrix:
+      pylatest_conda_mkl:
+        DISTRIB: 'conda'
+        PYTHON_VERSION: '*'
+        INSTALL_MKL: 'true'
+        NUMPY_VERSION: '*'
+        SCIPY_VERSION: '*'
+        CYTHON_VERSION: '*'
+        PILLOW_VERSION: '*'
+        PYTEST_VERSION: '*'
+        JOBLIB_VERSION: '*'
+        COVERAGE: 'true'
+
 - template: build_tools/azure/posix.yml
   parameters:
     name: Linux
     vmImage: ubuntu-16.04
+    dependsOn: [linting]
     matrix:
       # Linux environment to test that scikit-learn can be built against
       # versions of numpy, scipy with ATLAS that comes with Ubuntu Xenial 16.04
@@ -38,22 +78,12 @@ jobs:
         CHECK_PYTEST_SOFT_DEPENDENCY: 'true'
         TEST_DOCSTRINGS: 'true'
         CHECK_WARNINGS: 'true'
-      pylatest_conda_mkl:
-        DISTRIB: 'conda'
-        PYTHON_VERSION: '*'
-        INSTALL_MKL: 'true'
-        NUMPY_VERSION: '*'
-        SCIPY_VERSION: '*'
-        CYTHON_VERSION: '*'
-        PILLOW_VERSION: '*'
-        PYTEST_VERSION: '*'
-        JOBLIB_VERSION: '*'
-        COVERAGE: 'true'
 
 - template: build_tools/azure/posix-32.yml
   parameters:
     name: Linux32
     vmImage: ubuntu-16.04
+    dependsOn: [linting]
     matrix:
       py35_ubuntu_atlas_32bit:
         DISTRIB: 'ubuntu-32'
@@ -65,6 +95,7 @@ jobs:
   parameters:
     name: macOS
     vmImage: xcode9-macos10.13
+    dependsOn: [linting]
     matrix:
       pylatest_conda_mkl:
         DISTRIB: 'conda'
@@ -82,6 +113,7 @@ jobs:
   parameters:
     name: Windows
     vmImage: vs2017-win2016
+    dependsOn: [linting]
     matrix:
       py37_conda_mkl:
         PYTHON_VERSION: '3.7'
diff --git a/build_tools/azure/posix-32.yml b/build_tools/azure/posix-32.yml
index 127630b61ca65..606b2502daab3 100644
--- a/build_tools/azure/posix-32.yml
+++ b/build_tools/azure/posix-32.yml
@@ -2,9 +2,11 @@ parameters:
   name: ''
   vmImage: ''
   matrix: []
+  dependsOn: []
 
 jobs:
 - job: ${{ parameters.name }}
+  dependsOn: ${{ parameters.dependsOn }}
   pool:
     vmImage: ${{ parameters.vmImage }}
   variables:
diff --git a/build_tools/azure/posix.yml b/build_tools/azure/posix.yml
index 13bce4963cae9..d525214b3150b 100644
--- a/build_tools/azure/posix.yml
+++ b/build_tools/azure/posix.yml
@@ -2,9 +2,11 @@ parameters:
   name: ''
   vmImage: ''
   matrix: []
+  dependsOn: []
 
 jobs:
 - job: ${{ parameters.name }}
+  dependsOn: ${{ parameters.dependsOn }}
   pool:
     vmImage: ${{ parameters.vmImage }}
   variables:
diff --git a/build_tools/azure/windows.yml b/build_tools/azure/windows.yml
index e5a1eaf5fd9ce..76efdd54f83eb 100644
--- a/build_tools/azure/windows.yml
+++ b/build_tools/azure/windows.yml
@@ -3,9 +3,11 @@ parameters:
   name: ''
   vmImage: ''
   matrix: []
+  dependsOn: []
 
 jobs:
 - job: ${{ parameters.name }}
+  dependsOn: ${{ parameters.dependsOn }}
   pool:
     vmImage: ${{ parameters.vmImage }}
   variables:


Code review comments:
diff_hunk: @@ -1,9 +1,49 @@
 # Adapted from https://github.com/pandas-dev/pandas/blob/master/azure-pipelines.yml
 jobs:
+- job: linting
+  displayName: Linting
+  pool:
+    vmImage: ubuntu-16.04
+  steps:
+    - bash: echo "##vso[task.prependpath]$CONDA/bin"
+      displayName: Add conda to PATH
+    - bash: sudo chown -R $USER $CONDA
+      displayName: Take ownership of conda installation
+    - bash: conda create --name flake8_env --yes flake8
+      displayName: Install flake8
+    - bash: |
+        source activate flake8_env
+        ./build_tools/circle/flake8_diff.sh
path: azure-pipelines.yml
text: @user1:
You have two lint commands here... Should these be together in a single script to make it easy to change both circle and Azure runs?

@author:
I would be okay with combining them. What do you think of naming the combined script "linting.sh"?

@user2:
> I would be okay with combining them. What do you think of naming the combined script "linting.sh"?

sounds good to me.