Repository: dask/dask
Pull request: Allow the CI dev build to fail
Closes https://github.com/dask/dask/issues/3043
Partially addresses https://github.com/dask/dask/issues/2744

As dev libraries tend to breaking things rather frequently, simply mark this build as an allowed failure. This doesn't mean that we don't care about fixing issues with dev libraries. Simply that we don't expect every change to take that into account. Leaving it in the matrix for those interested in fixing it to have an easier time doing so.

Problem statement:
Allow failures in development tests
Currently one of the travis-ci configurations tests against master versions of NumPy, Pandas, and other libraries.  This has caused Dask testing builds to fail a couple of times recently.  Do we want to consider changing our travis-ci settings to allow this testing configuration to fail short term?

Hints from issue discussion:
I would prefer leaving it as for now (now that https://github.com/dask/dask/issues/3039 is fixed)

When something upstream does break dask, a maintainer could quickly make an issue calling attention to it and a PR [allowing failures](https://docs.travis-ci.com/user/customizing-the-build/#Rows-that-are-Allowed-to-Fail) on that build until its fixed.
IMHO this is a reasonable thing to do.

Development versions of libraries will inevitably introduce breaks that will cause failures. These are things that we certainly need to address, but not typically on the same time scale as issues with current stable releases. Not to mention it seems (to me at least) that problems in development libraries have been on the rise. In my rough estimation, it seems there has been an issue like this once a week. The consequence is reasonable PRs that have relatively little (or in some case nothing to do) with these development libraries end up being blocked rather frequently. Some of these include bug fixes for Dask or with stable library versions. Marking the development library build as a known failure IMHO more appropriately prioritizes these issues and allows development to continue at a more reasonable pace.

I suppose the counter to that would be that marking the development library build as an allowed failure could allow more issues to creep in. Maybe there is a way for those interested or closely associated with these development libraries to subscribe to these failures. Perhaps that would alleviate some of the concern about issues creeping in. Other thoughts on how to make this less burdensome?
Yep, I'd favor moving the development versions to a nightly cron job.

On Sat, Jan 20, 2018 at 2:07 PM, jakirkham <notifications@github.com> wrote:

> IMHO this is a reasonable thing to do.
>
> Development versions of libraries will inevitably introduce breaks that
> will cause failures. These are things that we certainly need to address,
> but not typically on the same time scale as issues with current stable
> releases. Not to mention it seems (to me at least) that problems in
> development libraries have been on the rise. In my rough estimation, it
> seems there has been an issue like this once a week. The consequence is
> reasonable PRs that have relatively little (or in some case nothing to do)
> with these development libraries end up being blocked rather frequently.
> Some of these include bug fixes for Dask or with stable library versions.
> Marking the development library build as a known failure IMHO more
> appropriately prioritizes these issues and allows development to continue
> at a more reasonable pace.
>
> I suppose the counter to that would be that marking the development
> library build as an allowed failure could allow more issues to creep in.
> Maybe there is a way for those interested or closely associated with these
> development libraries to subscribe to these failures. Perhaps that would
> alleviate some of the concern about issues creeping in. Other thoughts on
> how to make this less burdensome?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/3043#issuecomment-359198493>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIqRadg1rPM-ktuWRDrLEAOig_Z-Lks5tMkdzgaJpZM4RQ2HL>
> .
>

Put in PR ( https://github.com/dask/dask/pull/3088 ) to allow the dev build to fail for now. It certainly can be used to fix this issue if we agree with the change. Though it could also be used to discuss exactly how we would like to test development versions of dependencies going forward.

Commit message: Use newest NumPy and Pandas with Python 3.6

Make sure to use the newest stable releases of NumPy and Pandas in the
Python 3.6 build. That way we get the most benefit out of it.

Patch to review:
diff --git a/.travis.yml b/.travis.yml
index 8adc9583c93..1843d5c47d2 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -47,20 +47,29 @@ jobs:
       - *no_optimize
       - *no_imports
 
-    - env: &py36_env
+    - env:
       - PYTHON=3.6
+      - NUMPY=1.14.0
+      - PANDAS=0.22.0
+      - *no_coverage
+      - *no_optimize
+      - *imports
+
+    - env: &py36_dev
       - UPSTREAM_DEV=1  # Install nightly versions of NumPy, pandas, pyarrow
       - NUMPY=1.13.0    # these are overridden later
       - PANDAS=0.20.3
       - *no_coverage
       - *no_optimize
       - *imports
+      if: type != pull_request
 
-    - env: *py36_env
+    - env: *py36_dev
       if: type != pull_request
       os: osx
 
   allow_failures:
+    - env: *py36_dev
     - os: osx
 
 


Code review comments:
diff_hunk: @@ -47,20 +47,29 @@ jobs:
       - *no_optimize
       - *no_imports
 
-    - env: &py36_env
+    - env:
       - PYTHON=3.6
+      - NUMPY=1.14.0
+      - PANDAS=0.22.0
+      - *no_coverage
+      - *no_optimize
+      - *imports
+
+    - env: &py36_dev
       - UPSTREAM_DEV=1  # Install nightly versions of NumPy, pandas, pyarrow
       - NUMPY=1.13.0    # these are overridden later
       - PANDAS=0.20.3
       - *no_coverage
       - *no_optimize
       - *imports
+      if: type != pull_request
 
-    - env: *py36_env
+    - env: *py36_dev
path: .travis.yml
text: @user1:
I think we should use released versions on the osx tests instead. This used to be true, and then it wasn't when the dev tests were added. Ideally each split would test a different issue - the osx tests are for anything oddly osx specific. Adding in dev dependencies conflates things.

My preference would be the osx env matches the env you just added (with numpy=1.14.0).

@author:
Agreed. Have switched macOS to the latest stable releases as suggested.