Repository: scikit-learn/scikit-learn
Pull request: CI Adds quicker CI failure to reduce resource usage
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Closes https://github.com/scikit-learn/scikit-learn/issues/21403

#### What does this implement/fix? Explain your changes.
This PR allows `Ubuntu_Bionic` to run first and have most of the other instances depend on it.

1. I added a second `Ubuntu_Bionic_Parallel` that runs only when `[azure all-parallel]` is in the commit message. If `[azure all-parallel]` is not in the commit message, then `Ubuntu_Bionic_Parallel` is skipped.

2. `Ubuntu_Bionic` only runs when `[azure all-parallel]` is **not** in the commit message.

3. The other instances are updated to run when `Ubuntu_Bionic` succeeds or skipped, which covers all the non-failure cases.


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->


Problem statement:
Quicker Failing CI with reduced resource usage
Here is an idea to reduce our Azure Pipelines CI usage while trying to make it give informative feedback as possible in case of typos committed in PRs.

The suggestion would be to change the Azure Pipelines config to have 2 top level jobs:

- 1 linux quick-failing build that runs the tests with `pytest --maxfail=3`
- 1 linting job that runs flake8, mypy and black

Then if both those jobs complete successfully, run the other jobs to cover all the supported platforms.

The goal of this design would be to:

- to not waste too much resources for PRs that receive many sequential commits as most test failures are expected to be highly correlated across platforms,
- to still get informative informative test failure message to iterate, even if the linter fails for an unused import (already the case with the current config, this would be preserved),
- to not waste time in the first run with a very long failure report when half of the tests to fail for the same typo.

The downside it that when one expects the PR to be green, the total time will be "first test run + longest test run of the remainder" instead of just "linter time + longest test run" as is the case now with more parallelism.

Hints from issue discussion:
I like this approach overall.

> The downside it that when one expects the PR to be green, the total time will be "first test run + longest test run of the remainder" instead of just "linter time + longest test run" as is the case now with more parallelism.

We can have another commit tag `[azure all-parallel]`, to trigger "Run everything in parallel".
> We can have another commit tag [azure all-parallel], to trigger "Run everything in parallel".

That's a good idea but I don't know how to do it ;)
Also we should make sure that the fail-fast linux job also installs fast, in particular, does not install large dependencies like MKL and runs without coverage.

Commit message: Merge remote-tracking branch 'upstream/main' into ci_failing_faster

Patch to review:
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
index 1c9339308a45a..c4b2e2fcf1028 100644
--- a/azure-pipelines.yml
+++ b/azure-pipelines.yml
@@ -167,16 +167,18 @@ jobs:
         SKLEARN_TESTS_GLOBAL_RANDOM_SEED: '42'  # default global random seed
 
 # Check compilation with Ubuntu bionic 18.04 LTS and scipy from conda-forge
-- template: build_tools/azure/posix.yml
+- template: build_tools/azure/posix-all-parallel.yml
   parameters:
     name: Ubuntu_Bionic
     vmImage: ubuntu-18.04
-    dependsOn: [git_commit, linting]
+    dependsOn: [git_commit]
+    dependsOnParallel: [linting, git_commit]
     condition: |
       and(
         succeeded(),
         not(contains(dependencies['git_commit']['outputs']['commit.message'], '[ci skip]'))
       )
+    commitMessage: dependencies['git_commit']['outputs']['commit.message']
     matrix:
       py38_conda_forge_openblas_ubuntu_1804:
         DISTRIB: 'conda'
@@ -189,10 +191,10 @@ jobs:
   parameters:
     name: Linux
     vmImage: ubuntu-20.04
-    dependsOn: [linting, git_commit]
+    dependsOn: [linting, git_commit, Ubuntu_Bionic]
     condition: |
       and(
-        succeeded(),
+        in(dependencies['Ubuntu_Bionic']['result'], 'Succeeded', 'Skipped'),
         not(contains(dependencies['git_commit']['outputs']['commit.message'], '[ci skip]'))
       )
     matrix:
@@ -225,10 +227,10 @@ jobs:
   parameters:
     name: Linux_Docker
     vmImage: ubuntu-20.04
-    dependsOn: [linting, git_commit]
+    dependsOn: [linting, git_commit, Ubuntu_Bionic]
     condition: |
       and(
-        succeeded(),
+        in(dependencies['Ubuntu_Bionic']['result'], 'Succeeded', 'Skipped'),
         not(contains(dependencies['git_commit']['outputs']['commit.message'], '[ci skip]'))
       )
     matrix:
@@ -244,10 +246,10 @@ jobs:
   parameters:
     name: macOS
     vmImage: macOS-10.15
-    dependsOn: [linting, git_commit]
+    dependsOn: [linting, git_commit, Ubuntu_Bionic]
     condition: |
       and(
-        succeeded(),
+        in(dependencies['Ubuntu_Bionic']['result'], 'Succeeded', 'Skipped'),
         not(contains(dependencies['git_commit']['outputs']['commit.message'], '[ci skip]'))
       )
     matrix:
@@ -266,10 +268,10 @@ jobs:
   parameters:
     name: Windows
     vmImage: windows-latest
-    dependsOn: [linting, git_commit]
+    dependsOn: [linting, git_commit, Ubuntu_Bionic]
     condition: |
       and(
-        succeeded(),
+        in(dependencies['Ubuntu_Bionic']['result'], 'Succeeded', 'Skipped'),
         not(contains(dependencies['git_commit']['outputs']['commit.message'], '[ci skip]'))
       )
     matrix:
diff --git a/build_tools/azure/posix-all-parallel.yml b/build_tools/azure/posix-all-parallel.yml
new file mode 100644
index 0000000000000..534b5846ee34c
--- /dev/null
+++ b/build_tools/azure/posix-all-parallel.yml
@@ -0,0 +1,39 @@
+parameters:
+  name: ''
+  vmImage: ''
+  matrix: []
+  dependsOn: []
+  dependsOnParallel: []
+  condition: ''
+  commitMessage: ''
+
+jobs:
+# When [azure parallel] is *not* in the commit message, this build is enabled.
+# Later builds should check if this job succeeded or skipped by using:
+# in(dependencies[{{ parameters.name }}]['result'], 'Succeeded', 'Skipped')
+- template: posix.yml
+  parameters:
+    name: ${{ parameters.name }}
+    vmImage: ${{ parameters.vmImage }}
+    matrix: ${{ parameters.matrix }}
+    dependsOn: ${{ parameters.dependsOn }}
+    condition: |
+      and(
+        ${{ parameters.condition }},
+        not(contains(${{ parameters.commitMessage }}, '[azure parallel]'))
+      )
+
+# This configures the "parallel" build. The build on top is skipped so
+# later builds can run in parallel with this "parallel" configuration.
+# When [azure parallel] is in the commit message, this build is enabled.
+- template: posix.yml
+  parameters:
+    name: ${{ parameters.name }}_Parallel
+    vmImage: ${{ parameters.vmImage }}
+    matrix: ${{ parameters.matrix }}
+    dependsOn: ${{ parameters.dependsOnParallel }}
+    condition: |
+      and(
+        ${{ parameters.condition }},
+        contains(${{ parameters.commitMessage }}, '[azure parallel]')
+      )
diff --git a/doc/developers/contributing.rst b/doc/developers/contributing.rst
index 55ff67436a462..2b9af7442a4a1 100644
--- a/doc/developers/contributing.rst
+++ b/doc/developers/contributing.rst
@@ -551,6 +551,7 @@ message, the following actions are taken.
     [nogil]                Build & test with the nogil experimental branches of CPython, Cython, NumPy, SciPy...
     [icc-build]            Build & test with the Intel C compiler (ICC)
     [pypy]                 Build & test with PyPy
+    [azure parallel]   Run Azure CI in parallel
     [doc skip]             Docs are not built
     [doc quick]            Docs built, but excludes example gallery plots
     [doc build]            Docs built including example gallery plots (very long)


Code review comments:
diff_hunk: @@ -0,0 +1,39 @@
+parameters:
+  name: ''
+  vmImage: ''
+  matrix: []
+  dependsOn: []
+  dependsOnParallel: []
+  condition: ''
+  commitMessage: ''
+
+jobs:
+# When [azure parallel] is *not* in the commit message, this build is enabled.
+# Later builds should check if this job succeeded or skipped by using:
+# in(dependencies[{{ parameters.name }}]['result'], 'Succeeded', 'Skipped')
+- template: posix.yml
+  parameters:
+    name: ${{ parameters.name }}
+    vmImage: ${{ parameters.vmImage }}
+    matrix: ${{ parameters.matrix }}
+    dependsOn: ${{ parameters.dependsOn }}
+    condition: |
+      and(
+        ${{ parameters.condition }},
+        not(contains(${{ parameters.commitMessage }}, '[azure parallel]'))
+      )
+
+# This configures the "parallel" build. The build on top is skipped so
+# later builds can run in parallel with this "parallel" configuration.
+# When [azure parallel] is in the commit message, this build is enabled.
path: build_tools/azure/posix-all-parallel.yml
text: @user1:
Continuing the previous suggestion.
```suggestion
# When [azure parallel] is in the commit message, parallel runs are
# performed via this jobs' template parametrisation.
# The job on top is skipped so later jobs can be run in parallel
# with this jobs' template parametrisation.
```

diff_hunk: @@ -551,6 +551,7 @@ message, the following actions are taken.
     [nogil]                Build & test with the nogil experimental branches of CPython, Cython, NumPy, SciPy...
     [icc-build]            Build & test with the Intel C compiler (ICC)
     [pypy]                 Build & test with PyPy
+    [azure parallel]   Run Azure CI in parallel
path: doc/developers/contributing.rst
text: @user1:
```suggestion
    [pypy]                 Build & test with PyPy
    [azure parallel]       Run Azure CI jobs in parallel
```