Repository: ESMCI/cime

Issue:
Charting a path towards dropping python2 support
This question came up in yesterday's CESM Software Engineering Group meeting (raised by @gold2718 , also relevant to @alperaltuntas ): We have some python code coming into CESM that will assume python3. Some of this code will be invoked by cime scripts (buildnml / buildlib), so will run under whatever python version is used to invoke the cime scripts, as controlled by your environment together with the shebang lines in the various cime scripts. Our feeling in CESM is that it is reasonable at this point to require python3, since python2 support ended over a year ago.

**I guess the first question here is: Do others (e.g., in E3SM) agree that it's reasonable to require python3 moving forward?**

Currently, our shebang lines are `#!/usr/bin/env python`. My understanding (based on https://stackoverflow.com/questions/6908143/should-i-put-shebang-in-python-scripts-and-what-form-should-it-take, https://www.python.org/dev/peps/pep-0394/, etc.) is that that is correct usage when your python code is compatible with python2 and python3. However, some systems (like cheyenne) have `python` pointing to `python2`, even if you have loaded a python3 module, and it may be a while before we can assume `python` always refers to `python3`.

So I'd like to start a discussion about how we want to proceed here. I'm going under the assumption that all users will have python3 installed on their system, so the question is: **How do we ensure that the cime scripts are run under python3 rather than python2?**

Two possible solutions came up in our CSEG discussion:

1. Change our shebang lines to `#!/usr/bin/env python3`. My understanding (e.g., from the links a couple of paragraphs above) is that this is a generally recommended solution when you require python3. I'm not 100% sure that this is 100% reliable in all situations, though: it sounds to me like systems *should* define a python3 executable, but there may be some that don't.

2. Instruct users to use a python environment where `python` points to a python3 executable. From PEP 0394, it sounds like this will be the case if people use a python virtual environment. I have been wary of requiring use of a python virtual environment, because it feels like something that might make porting to a new machine more difficult for non-tech-savvy users, and might be hard for users to remember even on systems with tech-savvy sys admins: Users would need to remember to launch the right python environment before invoking any cime scripts (create_newcase, case.build, etc., etc., etc.).

My gut feeling leans towards (1), because part of the value of cime has been that it manages the environment for you, so (2) feels like it might add too much friction for the use of cime. But I'd like to hear thoughts from others, especially others who have more experience than I do with python system configuration & virtual environments (which is a low bar to be honest). One possible advantage of (2) is that it could help give a path towards requiring a limited number of python packages outside the standard library (for CESM, we've talked about yaml, numpy, and either scipy.io.netcdf or netcdf4).

Thoughts?


Discussion:
Definitely option 1.  
From cime telecon: People are fine with option (1). Will need to remove travis-ci python2 tests.

Then we can go through and do some cleanup where we were doing things for compatibility with python2 (including six library). That can be done as a later step. Another thing that would be nice (from @jgfouca ) is switching to pathlib.Path.
travis-ci was already removed from master.

Resolving patch:
diff --git a/scripts/Tools/generate_cylc_workflow.py b/scripts/Tools/generate_cylc_workflow.py
--- a/scripts/Tools/generate_cylc_workflow.py
+++ b/scripts/Tools/generate_cylc_workflow.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 """
 Generates a cylc workflow file for the case.  See https://cylc.github.io for details about cylc
diff --git a/scripts/Tools/testreporter.py b/scripts/Tools/testreporter.py
--- a/scripts/Tools/testreporter.py
+++ b/scripts/Tools/testreporter.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 """
 Simple script to populate CESM test database with test results.


Test patch:
diff --git a/scripts/Tools/bless_test_results b/scripts/Tools/bless_test_results
--- a/scripts/Tools/bless_test_results
+++ b/scripts/Tools/bless_test_results
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 """
 Analyze results from a test root area, finding namelist and non-BFB
diff --git a/scripts/Tools/compare_test_results b/scripts/Tools/compare_test_results
--- a/scripts/Tools/compare_test_results
+++ b/scripts/Tools/compare_test_results
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 """Analyze results from a test root area, comparing non-BFB changes.
 
diff --git a/scripts/Tools/component_compare_test b/scripts/Tools/component_compare_test
--- a/scripts/Tools/component_compare_test
+++ b/scripts/Tools/component_compare_test
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 """
 Compares two component history files in the testcase directory
diff --git a/scripts/Tools/list_e3sm_tests b/scripts/Tools/list_e3sm_tests
--- a/scripts/Tools/list_e3sm_tests
+++ b/scripts/Tools/list_e3sm_tests
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 """
 List e3sm test suites. Can be used to show what's being tested. Can just
diff --git a/scripts/Tools/wait_for_tests b/scripts/Tools/wait_for_tests
--- a/scripts/Tools/wait_for_tests
+++ b/scripts/Tools/wait_for_tests
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 """
 Wait for a queued set of E3SM tests to finish by watching the
diff --git a/scripts/create_test b/scripts/create_test
--- a/scripts/create_test
+++ b/scripts/create_test
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 """
 Script to create, build and run CIME tests. This script can:
diff --git a/scripts/tests/list_tests b/scripts/tests/list_tests
--- a/scripts/tests/list_tests
+++ b/scripts/tests/list_tests
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # This script will print the list of test classes in
 # scripts_regression_tests.py
 #
diff --git a/scripts/tests/scripts_regression_tests.py b/scripts/tests/scripts_regression_tests.py
--- a/scripts/tests/scripts_regression_tests.py
+++ b/scripts/tests/scripts_regression_tests.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 """
 Script containing CIME python regression test suite. This suite should be run
@@ -586,7 +586,7 @@ def test_f_createnewcase_with_user_compset(self):
         cls._testdirs.append(testdir)
 
         if CIME.utils.get_model() == "cesm":
-            # Will need to be updated when cesm brings in new share repo new line will be. 
+            # Will need to be updated when cesm brings in new share repo new line will be.
             # pesfile = os.path.join(get_src_root(),"cpl7","driver","cime_config","config_pes.xml")
             # or
             # pesfile = os.path.join(get_src_root(),"components","cmeps","cime_config","config_pes.xml")
@@ -1043,14 +1043,14 @@ def test_wait_for_test_wait_for_pend(self):
 
         time.sleep(5) # Kinda hacky
 
-        self.assertTrue(run_thread.isAlive(), msg="wait_for_tests should have waited")
+        self.assertTrue(run_thread.is_alive(), msg="wait_for_tests should have waited")
 
         with TestStatus(test_dir=os.path.join(self._testdir_unfinished, "5")) as ts:
             ts.set_status(RUN_PHASE, TEST_PASS_STATUS)
 
         run_thread.join(timeout=10)
 
-        self.assertFalse(run_thread.isAlive(), msg="wait_for_tests should have finished")
+        self.assertFalse(run_thread.is_alive(), msg="wait_for_tests should have finished")
 
         self.assertTrue(self._thread_error is None, msg="Thread had failure: %s" % self._thread_error)
 
@@ -1063,14 +1063,14 @@ def test_wait_for_test_wait_for_missing_run_phase(self):
 
         time.sleep(5) # Kinda hacky
 
-        self.assertTrue(run_thread.isAlive(), msg="wait_for_tests should have waited")
+        self.assertTrue(run_thread.is_alive(), msg="wait_for_tests should have waited")
 
         with TestStatus(test_dir=os.path.join(self._testdir_unfinished2, "5")) as ts:
             ts.set_status(RUN_PHASE, TEST_PASS_STATUS)
 
         run_thread.join(timeout=10)
 
-        self.assertFalse(run_thread.isAlive(), msg="wait_for_tests should have finished")
+        self.assertFalse(run_thread.is_alive(), msg="wait_for_tests should have finished")
 
         self.assertTrue(self._thread_error is None, msg="Thread had failure: %s" % self._thread_error)
 
@@ -1084,13 +1084,13 @@ def test_wait_for_test_wait_kill(self):
 
         time.sleep(5)
 
-        self.assertTrue(run_thread.isAlive(), msg="wait_for_tests should have waited")
+        self.assertTrue(run_thread.is_alive(), msg="wait_for_tests should have waited")
 
         kill_python_subprocesses(signal.SIGTERM, expected_num_killed=1, tester=self)
 
         run_thread.join(timeout=10)
 
-        self.assertFalse(run_thread.isAlive(), msg="wait_for_tests should have finished")
+        self.assertFalse(run_thread.is_alive(), msg="wait_for_tests should have finished")
 
         self.assertTrue(self._thread_error is None, msg="Thread had failure: %s" % self._thread_error)
 
@@ -1106,7 +1106,7 @@ def test_wait_for_test_cdash_pass(self):
 
         run_thread.join(timeout=10)
 
-        self.assertFalse(run_thread.isAlive(), msg="wait_for_tests should have finished")
+        self.assertFalse(run_thread.is_alive(), msg="wait_for_tests should have finished")
 
         self.assertTrue(self._thread_error is None, msg="Thread had failure: %s" % self._thread_error)
 
@@ -1124,13 +1124,13 @@ def test_wait_for_test_cdash_kill(self):
 
         time.sleep(5)
 
-        self.assertTrue(run_thread.isAlive(), msg="wait_for_tests should have waited")
+        self.assertTrue(run_thread.is_alive(), msg="wait_for_tests should have waited")
 
         kill_python_subprocesses(signal.SIGTERM, expected_num_killed=1, tester=self)
 
         run_thread.join(timeout=10)
 
-        self.assertFalse(run_thread.isAlive(), msg="wait_for_tests should have finished")
+        self.assertFalse(run_thread.is_alive(), msg="wait_for_tests should have finished")
         self.assertTrue(self._thread_error is None, msg="Thread had failure: %s" % self._thread_error)
 
         assert_dashboard_has_build(self, build_name)
@@ -1161,7 +1161,7 @@ def live_test_impl(self, testdir, expected_results, last_phase, last_status):
 
         time.sleep(5)
 
-        self.assertTrue(run_thread.isAlive(), msg="wait_for_tests should have waited")
+        self.assertTrue(run_thread.is_alive(), msg="wait_for_tests should have waited")
 
         for core_phase in CORE_PHASES[1:]:
             with TestStatus(test_dir=os.path.join(self._testdir_teststatus1, "0")) as ts:
@@ -1170,10 +1170,10 @@ def live_test_impl(self, testdir, expected_results, last_phase, last_status):
             time.sleep(5)
 
             if core_phase != last_phase:
-                self.assertTrue(run_thread.isAlive(), msg="wait_for_tests should have waited after passing phase {}".format(core_phase))
+                self.assertTrue(run_thread.is_alive(), msg="wait_for_tests should have waited after passing phase {}".format(core_phase))
             else:
                 run_thread.join(timeout=10)
-                self.assertFalse(run_thread.isAlive(), msg="wait_for_tests should have finished after phase {}".format(core_phase))
+                self.assertFalse(run_thread.is_alive(), msg="wait_for_tests should have finished after phase {}".format(core_phase))
                 break
 
         self.assertTrue(self._thread_error is None, msg="Thread had failure: %s" % self._thread_error)
@@ -1637,7 +1637,7 @@ def test_jenkins_generic_job_kill(self):
 
         run_thread.join(timeout=30)
 
-        self.assertFalse(run_thread.isAlive(), msg="jenkins_generic_job should have finished")
+        self.assertFalse(run_thread.is_alive(), msg="jenkins_generic_job should have finished")
         self.assertTrue(self._thread_error is None, msg="Thread had failure: %s" % self._thread_error)
         assert_dashboard_has_build(self, build_name)