Repository: spack/spack

Issue:
Python3 Migration Strategy
@liamrpowell @adamjstewart @tgamblin @alalazo @scheibelp 
Comments / discussion encouraged.  We need to get this figured out, and we're running out of time.

Problem Summary
=============

In just 7.5 months (2020-01-01), Python2 will be *officially retired*; and Spack currently lacks a coherent plan of how we will deal with this.  At that point, a number of well-known Python libraries will *require* Python3.  If Spack cannot easily install Python3 stacks *out of the box* by then, I believe we stand to lose users to systems that CAN do that (eg Anaconda); see #11396.  I believe that January 1 2020 is the date that we should *officially* switch Spack to default to Python3.

The purpose of this thread is to come up with a roadmap on how to do so.  The core problem is that, with the current concretizer, it is not possible to have packages support Python2 and Python3 in a totally transparent manner.  For example, in `py-mypackage.py` we need to (but cannot use) statements such as:
```
depends_on('py-backported_stuff', when='^python@2')
```

General Solutions
=============

This problem has prevented Spack from supporting Python3 in a "just works" manner, so far.  There are two known solutions to it:

1. #7926 uses a `+python3` variant as a manual way for users to specify that Python3 is being used.  This variant is applied to app packages (where applicable) using the `all:` functionality in `packages.yaml`.  The above dependency can then be rewritten as:
```
depends_on('py-backported_stuff', when='~python3')
```
A user working in Python3 land can pretty much set-and-forget the `all: +python3` directive in `packages.yaml`.

2. One might consider #7926 to be a dirty "hack."  A new concretizer that fixes these problem promises to be a more elegant long-term solution, allowing Python packages to specify dependencies directly.

So far, a long-term preference for (2) has prevented us from merging (1).  However, I believe the time to wait for an improved concretizer is over: We need to be ready for an all-Python3 world by the end of this year.  And currently, we are *NOT EVEN CLOSE*.  Therefore, I believe we should formulate a plan along the lines of (1), using technology we *already have*.  That will give us enough time to fix all the Python packages that need fixing, and to test them with Python3.

Python3 Transition Plan
==================

This is a transition plan that I think makes sense.

1. Re-work #7926 to use a `python2` variant instead of `python3`.  This makes sense because starting 2020, Python3 needs to be the default, in accordance with the rest of the Python community.

2. Review all Python packages for possible Python2-vs-3 issues; and code them in appropriately.  Use the examples in #7926 as a guide.  Also see #10319.

3. Test that `spack install py-xyz` works for all Python packages.  Test "out of the box" (Python3), and also with the `all: +python2` variant enabled.  We expect that some packages might not work for Python2 or Python3; and they should be updated appropriately.   Sometimes a newer upstream version might be needed; in other cases, a package might really be Python2-only and be headed for the dustbin.  Do this all on a branch that's periodically kept up-to-date with Spack's `develop`.

4. Fall 2019: Provide ABUNDANT notification to the Spack community that starting January 1 2020, Spack will go Python3 by default.  Provide instructions on how to continue building Python2 stacks after that date.

5. Tag the last version of Spack BEFORE Python3 is made default; and notify users of that tag, if they wish to keep using it for a while (without re-doing their config to continue using Python2).

6. January 1 2020: Merge the branch into Spack's `develop`.

7. When the new concretizer comes out, remove the `+python2` hack.

Alternate Plan
--------------

The above plan maintains backwards compatibility with Python2 systems.  Alternately, we could decide that this is too much effort.  In that case, the branch we prepare would only be tested for Python3, and the `+python2` variant would not be needed.  Python2-specific dependencies could simply be eliminated from Packages, making them simpler.

The alternate plan would be less work.  But it would also mean that Spack users still using Python2 would no longer be able to upgrade to the latest Spack.  My guess is, the additional effort to maintain backwards compatibility is probably worth it.



Discussion:

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/python/package.py b/var/spack/repos/builtin/packages/python/package.py
--- a/var/spack/repos/builtin/packages/python/package.py
+++ b/var/spack/repos/builtin/packages/python/package.py
@@ -32,7 +32,7 @@ class Python(AutotoolsPackage):
     maintainers = ['adamjstewart']
 
     version('3.8.0',  sha256='f1069ad3cae8e7ec467aa98a6565a62a48ef196cb8f1455a245a08db5e1792df')
-    version('3.7.4',  sha256='d63e63e14e6d29e17490abbe6f7d17afb3db182dbd801229f14e55f4157c4ba3')
+    version('3.7.4',  sha256='d63e63e14e6d29e17490abbe6f7d17afb3db182dbd801229f14e55f4157c4ba3', preferred=True)
     version('3.7.3',  sha256='d62e3015f2f89c970ac52343976b406694931742fbde2fed8d1ce8ebb4e1f8ff')
     version('3.7.2',  sha256='f09d83c773b9cc72421abba2c317e4e6e05d919f9bcf34468e192b6a6c8e328d')
     version('3.7.1',  sha256='36c1b81ac29d0f8341f727ef40864d99d8206897be96be73dc34d4739c9c9f06')
@@ -55,7 +55,7 @@ class Python(AutotoolsPackage):
     version('3.3.6',  sha256='0a58ad1f1def4ecc90b18b0c410a3a0e1a48cf7692c75d1f83d0af080e5d2034')
     version('3.2.6',  sha256='fc1e41296e29d476f696303acae293ae7a2310f0f9d0d637905e722a3f16163e')
     version('3.1.5',  sha256='d12dae6d06f52ef6bf1271db4d5b4d14b5dd39813e324314e72b648ef1bc0103')
-    version('2.7.16', sha256='01da813a3600876f03f46db11cc5c408175e99f03af2ba942ef324389a83bad5', preferred=True)
+    version('2.7.16', sha256='01da813a3600876f03f46db11cc5c408175e99f03af2ba942ef324389a83bad5')
     version('2.7.15', sha256='18617d1f15a380a919d517630a9cd85ce17ea602f9bbdc58ddc672df4b0239db')
     version('2.7.14', sha256='304c9b202ea6fbd0a4a8e0ad3733715fbd4749f2204a9173a58ec53c32ea73e8')
     version('2.7.13', sha256='a4f05a0720ce0fd92626f0278b6b433eee9a6173ddf2bced7957dfb599a5ece1')
@@ -145,6 +145,10 @@ class Python(AutotoolsPackage):
     # https://github.com/python/cpython/commit/0b91f8a668201fc58fa732b8acc496caedfdbae0
     patch('gcc-8-2.7.14.patch', when='@2.7.14 %gcc@8:')
 
+    # Fixes build with the Intel compilers
+    # https://github.com/python/cpython/pull/16717
+    patch('intel-3.6.7.patch', when='@3.6.7:3.6.8,3.7.1: %intel')
+
     # For more information refer to this bug report:
     # https://bugs.python.org/issue29712
     conflicts(
@@ -155,11 +159,6 @@ class Python(AutotoolsPackage):
     conflicts('+tix', when='~tkinter',
               msg='python+tix requires python+tix+tkinter')
 
-    # Python 3.6.7 and above can not be compiled with the Intel compiler
-    # https://bugs.python.org/issue35473
-    # https://bugs.python.org/issue37415
-    conflicts('%intel', when='@3.6.7:')
-
     _DISTUTIL_VARS_TO_SAVE = ['LDSHARED']
     _DISTUTIL_CACHE_FILENAME = 'sysconfig.json'
     _distutil_vars = None


Test patch: