Repository: dask/dask

Issue:
Timeline for dropping Python 2.7 support
Python 2 is being sunsetted by the PSF, and many projects are following suit: https://python3statement.org/.

Several of our dependencies (numpy, pandas, scikit-learn) and a few known dependents (xarray, scikit-learn) are ending support for Python 2 in new releases at the end of 2018. Scikit-image has already dropped support in new releases. We may want to follow suit.

A few questions:
- When should we end python 2.7 support? Is Jan 1 2019 too soon?
- Is this something that downstream projects will be fine with?
- Do we want to do a clean break (my vote) or continue with bugfix releases/LTS for python 2 for a while?


Discussion:
cc @mrocklin, @shoyer, @jakirkham, @TomAugspurger, @martindurant (and any others who may wish to comment).
Would recommend we look at our dependencies and see when those projects plan to drop Python 2 support as that will constrain how long a Python 2 versions of Dask packages would be viable. Also reaching out to downstream users to get an idea of how long they plan to support Python 2 would also be useful.
That's what I'm suggesting here.

> Several of our dependencies (numpy, pandas, scikit-learn) and a few known dependents (xarray, scikit-learn) are ending support for Python 2 in new releases at the end of 2018.

**Dependencies**: numpy, pandas, and scikit-learn are all dropping support at the end of the year
**Dependents**: scikit-learn and xarray are dropping support at the end of the year, scikit-image has already dropped support.

I don't know what other downstream packages rely on dask, but it would be good to consider those users as well.
To reiterate pros and cons.

There are multiple concrete benefits to consider, some of which are not very significant
- less developer time to make version-agnostic code
- possible faster code available in py3
- faster CI runs

I imagine that there will be users with high inertia and latency that won't update for a long time. They may be happy using old versions, but any bug in old code would be very painful if we don't have a LTS branch for fixes. I don't think we can suppose anymore that people are always happy to update their systems just because they want to use dask (unlike when dask was young and volatile). 
A very specific benefit might be using `async` within the Dask codebase (perhaps this is under item 2 in Martin's bullet points :). This could be very useful for making Dask-only (w/o Distributed) schedulers handle some of the more complex cases.
Some other projects that may be affected are `hyperspy`, `tensorflow` (has optional contrib support), `streamz`, `arrow`, and some `dask-*` (a few outside of the org).

cc @francisco-dlp @rmax @CJ-Wright @wesm
Arrow is only a dependency of `dask` and thus we're not affected by dask dropping Python 2. We have not yet made a clear statement towards dropping Python 2 in Arrow but I guess we should also follow along and do it at the same time as Pandas & NumPy does (but that will be decided on the Arrow mailing list soon).
You don't need to be concerned about TensorFlow. Last time I checked the part that uses Dask was deprecated.
This is not an issue for HyperSpy either, we dropped support for py27 in v.0.8.4 and that version didn't require dask.
> You don't need to be concerned about TensorFlow. Last time I checked the part that uses Dask was deprecated.

Yeah I saw a deprecation warning that was stating it was moved to a different part of TensorFlow and saw the new location seemed to be deprecation warning free. That said, was only skimming this code. So happy to trust your judgement. Just thought it was worth mentioning as I wasn't aware Tensorflow made use of Dask initially.
Related question, should Dask 1.0.0 support Python 2.7 or should that be when we drop it?
Here's the pip installs for Dask from PyPI for September 2018:

| python_version | percent | download_count |
| -------------- | ------: | -------------: |
| 3.6            |  83.14% |      1,352,891 |
| 2.7            |  11.18% |        181,892 |
| 3.5            |   4.35% |         70,744 |
| 3.7            |   0.77% |         12,564 |
| 3.4            |   0.55% |          8,877 |
| 2.6            |   0.02% |            261 |
| 3.3            |   0.00% |              3 |
| Total          |         |      1,627,232 |

Source: `pypinfo --start-date 2018-09-01 --end-date 2018-09-30 --percent --markdown dask pyversion`

11% for Python 2.7 is really quite low, numpy is at about 40%, pandas and scikit-learn at about 55%.
Such statistics should also be available for conda?

(also: who is using 2.6??)
Installs for conda for September 2018:

| python_version | percent | download_count |
| -------------- | ------: | -------------: |
| 2.7            |  31.75% |          4,441 |
| 3.5            |  26.70% |          3,735 |
| 3.6            |  24.87% |          3,478 |
| 3.7            |  14.46% |          2,022 |
| 3.4            |   2.11% |            295 |
| 2.6            |   0.10% |             14 |
| 3.3            |   0.01% |              2 |
| Total          |         |         13,987 |

Source: `pypinfo --start-date 2018-09-01 --end-date 2018-09-30 --percent --markdown conda pyversion`

(Some people are using old versions of Linux and don't want to or can't upgrade their Python.)
I meant installs of dask via conda :)  
Probably someone at Anaconda can get those numbers for us, although for a noarch package maybe it involves some work or perhaps is not available at all.
The data is definitely available within the Anaconda company, but currently
no public view exists.  This is an ongoing effort.  cc @seibert

On Sun, Oct 21, 2018 at 8:56 AM Martin Durant <notifications@github.com>
wrote:

> I meant installs of dask via conda :)
> Probably someone at Anaconda can get those numbers for us, although for a
> noarch package maybe it involves some work or perhaps is not available at
> all.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/4047#issuecomment-431666534>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AASszKLuKM2r4xIbpbE2rtqOyLvv_-uNks5unG8UgaJpZM4XEv1d>
> .
>

Most of the discussion here has been about downloads and dependencies in open source.  I think that we're missing the silent constituency of large organizations.  Institutions like corporations, government agencies, city governments, and so on are often not well represented in these numbers.  Sometimes they download the package once and then redistribute to thousands of users on their own.  Often they don't feel comfortable speaking up on open forums.  Often they have very large million-line codebases that are difficult to just "switch over".  We shouldn't necessarily be tied to Python 2 forever because of these users, but I do think that we should intentionally consider them as we make this decision.

I think that it would be good to get the perspective of @pzwang or others with more exposure to working with institutions.
I personally feel that while large organizations may move slower here, or be less likely to speak up, that shouldn't necessarily restrict us as maintainers without some kind of incentive from them. Dask currently works fine as is on Python 2.7, and those restricted from upgrading can continue to run the existing version. IMO their issues upgrading shouldn't impede us from moving forward unless they're paying someone to maintain py2 compatibility. I also think that with dependencies like numpy and pandas moving off of py2 our maintenance burden here will likely increase.

I do agree it would be good to get the perspective of @pzwang or others though.
In general I agree that they shouldn't restrict us fully

> without some kind of incentive from them
> unless they're paying someone to maintain py2 compatibility. 

They may indirectly pay for your and my salaries today.  The integration of OSS with business is a new thing that we're still all learning to deal with.  Jobs like ours sort of depend on further continued corporate engagement.  I think the active question is how companies will respond to the drop of Python 2.  

In an extreme situation where they decide "well screw it, lets just reinvent the stack in Go/Rust/..." or something like that then I think that dropping Python 2 would probably be a bad choice for the project because it would significantly reduce the number of people paid to work on it.

I think that the conservative choice is to wait until Numpy, Pandas, Scikit-Learn do a Python-3-only release, and then watch how companies respond.  I'm not saying that we should do that.  I'm saying that that's an option.  
In general, I think it's very good to be aware of the "silent but vast" number of commercial users that rely on the scipy/pydata stack, and who are stuck on Python 2.x (sometimes Python 2.5 or even 2.4.)  Such users are also oftentimes lurking on threads like these.

The question facing Dask is really the same one facing the entirety of the volunteer-based Python ecosystem.  Unfortunately, I am not aware of any easy answers here.  If I could wave a magic wand, I would like to see paid support being offered for 2.x maintenance for Scipy and PyData.  (Maybe this is something Enthought or Quansight can spearhead?)

The only wrinkle that's different for Dask than for, say, Numpy is due to Dask's relative newness.  This means that there are probably fewer large legacy 2.7 codebases with a heavy Dask dependency.  Additionally, because of Dask's relative youth, and the importance of scalable computing for Python, there is much larger opportunity cost of slowing down development by sinking (scarce) developer time into 2.x maintenance or compatibility.  (FWIW, I wouldn't make either statement about Numpy, where stability is paramount.)

So, from a practical standpoint, I think that having a usable 2.x LTS version of Dask, before a clean break, would be very, very good.  It would be even better to have a good commercial support alternative to suggest to commercial users stuck with Python 2, but that may be outside the scope of this community.  But my 2 cents is that you guys should solve for development velocity and innovation.

Works for me.  Thanks for chiming in Peter!
I propose to drop Python 2 support in the dask/distributed project at the end of the month.  Any objections?
Fine by me. I'd slightly prefer dropping 2.7 support from all projects at the same time (so we can say "dask releases after April 1, 2019 are Python 3 only", rather than "some dask projects support python 2, others don't"). Are you planning on making a final py27 release of distributed before removing support?
Some Dask projects are already Python 3 only, notably dask-kubernetes and dask-labextension.  From my perspective it has been interesting seeing the response to this.  There was some pushback on dask-kubernetes for example, which was interesting to observe.  

> Are you planning on making a final py27 release of distributed before removing support?

Yes.  I'm also happy to delay things.  
https://twitter.com/dask_dev/status/1108762527175852032
Some notes from the monthly Dask meeting last week.

-  People seem generally in favor of dropping soonish.  There were questions on how to drop.
    -  We might warn people, by placing a warning on import, similar to how `pip` does things.
    -  We might publish a package that is still technically Python 2 compatibile, but we publish it as Python 3 only and see if anyone cares
    -  We might make a trivially Python-2 incompatible release and see how things go, being willing to reverse it in the future
    - We might just go for it and drop Python 2 and stop worrying
-  It was brought up in a side conversation around the same time that after Core Python drops Python 2, other institutions may pick it up, and try to fork and move the codebase forward (2.8).  There are several large companies that might need to do this internally anyway.
-  On the dask.distributed side, intermittent CI failures in Python 2 are becoming increasingly problematic, which adds some time pressure, at least to that project.
I'm personally somewhat against having a warning.  From a usability perspective it's particularly loud, and difficult to work around if you're a novice user.  (My sense is that institutions that use Dask today will probably find this a large pain)

I'm in favor of dropping Python 2 in `dask/distributed` after the next `dask/distributed` release, and don't have strong thoughts on when to drop Python 2 in `dask/dask`.  I acknowledge the desire to drop things at the same time.  

I suggest that we drop Python 2 in `dask/distributed` soonish. but without making irreversible changes, and then make a formal announcement only when we drop things in `dask/dask`.  I think that this is a small and safe step that solves short term pain, and also gives us some information on if anyone cares.
> I suggest that we drop Python 2 in dask/distributed soonish. but without making irreversible changes, and then make a formal announcement only when we drop things in dask/dask.

That seems reasonable to me. I'm not sure how many will notice though - both pip and conda will just install the last valid version for py27, so unless the user explicitly asks for the new version they won't see an error.
Dropping Python 2 from CI in dask/distributed: https://github.com/dask/distributed/pull/2615
> I'm personally somewhat against having a warning. From a usability perspective it's particularly loud, and difficult to work around if you're a novice user.

So if not a warning, how do we ensure users know that Python 2 is no longer supported?

Alternatively if we were to add a warning, how would you configure the warning to make it easy to turn off?
> So if not a warning, how do we ensure users know that Python 2 is no longer supported?

Python 2 doesn't stop working, they just can't get new dask releases on Python 2.
> So if not a warning, how do we ensure users know that Python 2 is no longer supported?

I think that most projects just announce it on social media these days.  
> So if not a warning, how do we ensure users know that Python 2 is no longer supported?

Other ideas:
* put it prominently in the release notes, docs and README
* pin this issue
* sign https://python3statement.org

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -39,8 +39,6 @@
       license='BSD',
       keywords='task-scheduling parallel numpy pandas pydata',
       classifiers=[
-          "Programming Language :: Python :: 2",
-          "Programming Language :: Python :: 2.7",
           "Programming Language :: Python :: 3",
           "Programming Language :: Python :: 3.5",
           "Programming Language :: Python :: 3.6",
@@ -48,7 +46,7 @@
       ],
       packages=packages + tests,
       long_description=open('README.rst').read() if exists('README.rst') else '',
-      python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*",
+      python_requires=">=3.5",
       setup_requires=setup_requires,
       tests_require=['pytest'],
       extras_require=extras_require,


Test patch:
diff --git a/dask/dataframe/io/tests/test_parquet.py b/dask/dataframe/io/tests/test_parquet.py
--- a/dask/dataframe/io/tests/test_parquet.py
+++ b/dask/dataframe/io/tests/test_parquet.py
@@ -12,7 +12,7 @@
 import dask
 import dask.multiprocessing
 import dask.dataframe as dd
-from dask.dataframe.utils import assert_eq
+from dask.dataframe.utils import assert_eq, PANDAS_VERSION
 from dask.dataframe.io.parquet import _parse_pandas_metadata
 from dask.utils import natural_sort_key
 
@@ -476,8 +476,8 @@ def test_optimize(tmpdir, c):
     assert all(v[4] == c for v in dsk.values())
 
 
-@pytest.mark.skipif(not hasattr(pd.DataFrame, 'to_parquet'),
-                    reason="no to_parquet method")
+@pytest.mark.skipif(PANDAS_VERSION < '0.22.0',
+                    reason="new pyarrow assumes new-ish pandas versions")
 @write_read_engines()
 def test_roundtrip_from_pandas(tmpdir, write_engine, read_engine):
     fn = str(tmpdir.join('test.parquet'))