Repository: ESMCI/cime

Issue:
Failures in scripts_regression_tests K_TestCimeCase
K_TestCimeCase tests are failing on master, on yellowstone:

```
[yslogin3:/glade/p/work/sacks/cime_for_comparison/scripts/tests]$ ./scripts_regression_tests.py K_TestCimeCase

Testing commit 7c28d90d8b6701ef7b0d30b8805b6e1619992098
Using cime_model = cesm
Testing machine = yellowstone
Test root: /glade/scratch/sacks/scripts_regression_test.20170424_151039

test_cime_case (__main__.K_TestCimeCase) ... ok
test_cime_case_build_threaded_1 (__main__.K_TestCimeCase) ... ok
test_cime_case_build_threaded_2 (__main__.K_TestCimeCase) ... ok
test_cime_case_force_pecount (__main__.K_TestCimeCase) ... FAIL
Detected failed test or user request no teardown
Leaving files:
  /glade/scratch/sacks/scripts_regression_test.20170424_151039/cs.submit.fake_testing_only_20170424_151116
  /glade/scratch/sacks/scripts_regression_test.20170424_151039/cs.status.fake_testing_only_20170424_151116
  /glade/scratch/sacks/scripts_regression_test.20170424_151039/TESTRUNPASS_Mmpi-serial_P16x8.f19_g16_rx1.A.yellowstone_intel.fake_testing_only_20170424_151116
test_cime_case_mpi_serial (__main__.K_TestCimeCase) ... FAIL
Detected failed test or user request no teardown
Leaving files:
  /glade/scratch/sacks/scripts_regression_test.20170424_151039/TESTRUNPASS_Mmpi-serial.f19_g16_rx1.A.yellowstone_intel.fake_testing_only_20170424_151119
  /glade/scratch/sacks/scripts_regression_test.20170424_151039/cs.status.fake_testing_only_20170424_151119
  /glade/scratch/sacks/scripts_regression_test.20170424_151039/cs.submit.fake_testing_only_20170424_151119
test_cime_case_xmlchange_append (__main__.K_TestCimeCase) ... FAIL
Detected failed test or user request no teardown
Leaving files:
  /glade/scratch/sacks/scripts_regression_test.20170424_151039/cs.status.fake_testing_only_20170424_151121
  /glade/scratch/sacks/scripts_regression_test.20170424_151039/TESTRUNPASS_Mmpi-serial_P16x8.f19_g16_rx1.A.yellowstone_intel.fake_testing_only_20170424_151121
  /glade/scratch/sacks/scripts_regression_test.20170424_151039/cs.submit.fake_testing_only_20170424_151121

======================================================================
FAIL: test_cime_case_force_pecount (__main__.K_TestCimeCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./scripts_regression_tests.py", line 1373, in test_cime_case_force_pecount
    % (SCRIPT_DIR, self._baseline_name, self._testroot, self._testroot))
  File "./scripts_regression_tests.py", line 53, in run_cmd_assert_result
    test_obj.assertEqual(stat, expected_stat, msg=msg)
AssertionError:
COMMAND: /glade/p/work/sacks/cime_for_comparison/scripts/create_test TESTRUNPASS_Mmpi-serial.f19_g16_rx1.A -t fake_testing_only_20170424_151116 --no-build --test-root /glade/scratch/sacks/scripts_regression_test.20170424_151039 --output-root /glade/scratch/sacks/scripts_regression_test.20170424_151039 --force-procs 16 --force-threads 8
FROM_DIR: /glade/p/work/sacks/cime_for_comparison/scripts/tests
SHOULD HAVE WORKED, INSTEAD GOT STAT 100
OUTPUT: Testnames: ['TESTRUNPASS_Mmpi-serial.f19_g16_rx1.A.yellowstone_intel']
Using project from .cesm_proj: P93300601
Creating test directory /glade/scratch/sacks/scripts_regression_test.20170424_151039/TESTRUNPASS_Mmpi-serial_P16x8.f19_g16_rx1.A.yellowstone_intel.fake_testing_only_20170424_151116
RUNNING TESTS:
  TESTRUNPASS_Mmpi-serial_P16x8.f19_g16_rx1.A.yellowstone_intel
Starting CREATE_NEWCASE for test TESTRUNPASS_Mmpi-serial_P16x8.f19_g16_rx1.A.yellowstone_intel with 1 procs
Finished CREATE_NEWCASE for test TESTRUNPASS_Mmpi-serial_P16x8.f19_g16_rx1.A.yellowstone_intel in 1.863347 seconds (FAIL). [COMPLETED 1 of 1]
    Case dir: /glade/scratch/sacks/scripts_regression_test.20170424_151039/TESTRUNPASS_Mmpi-serial_P16x8.f19_g16_rx1.A.yellowstone_intel.fake_testing_only_20170424_151116
    Errors were:
        WARNING: No cesm Model version found.
        Traceback (most recent call last):
          File "/glade/p/work/sacks/cime_for_comparison/scripts/create_newcase", line 213, in <module>
            _main_func(__doc__)
          File "/glade/p/work/sacks/cime_for_comparison/scripts/create_newcase", line 196, in _main_func
            input_dir=input_dir)
          File "/glade/p/work/sacks/cime_for_comparison/scripts/Tools/../../scripts/lib/CIME/case.py", line 812, in configure
            self._initialize_derived_attributes()
          File "/glade/p/work/sacks/cime_for_comparison/scripts/Tools/../../scripts/lib/CIME/case.py", line 152, in _initialize_derived_attributes
            if "aprun" in executable:
        TypeError: argument of type 'NoneType' is not iterable

At test-scheduler close, state is:
FAIL TESTRUNPASS_Mmpi-serial_P16x8.f19_g16_rx1.A.yellowstone_intel (phase CREATE_NEWCASE)
    Case dir: /glade/scratch/sacks/scripts_regression_test.20170424_151039/TESTRUNPASS_Mmpi-serial_P16x8.f19_g16_rx1.A.yellowstone_intel.fake_testing_only_20170424_151116
test-scheduler took 2.00556492805 seconds
ERRPUT:


======================================================================
FAIL: test_cime_case_mpi_serial (__main__.K_TestCimeCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./scripts_regression_tests.py", line 1355, in test_cime_case_mpi_serial
    % (SCRIPT_DIR, self._baseline_name, self._testroot, self._testroot))
  File "./scripts_regression_tests.py", line 53, in run_cmd_assert_result
    test_obj.assertEqual(stat, expected_stat, msg=msg)
AssertionError:
COMMAND: /glade/p/work/sacks/cime_for_comparison/scripts/create_test TESTRUNPASS_Mmpi-serial.f19_g16_rx1.A -t fake_testing_only_20170424_151119 --no-build --test-root /glade/scratch/sacks/scripts_regression_test.20170424_151039 --output-root /glade/scratch/sacks/scripts_regression_test.20170424_151039
FROM_DIR: /glade/p/work/sacks/cime_for_comparison/scripts/tests
SHOULD HAVE WORKED, INSTEAD GOT STAT 100
OUTPUT: Testnames: ['TESTRUNPASS_Mmpi-serial.f19_g16_rx1.A.yellowstone_intel']
Using project from .cesm_proj: P93300601
Creating test directory /glade/scratch/sacks/scripts_regression_test.20170424_151039/TESTRUNPASS_Mmpi-serial.f19_g16_rx1.A.yellowstone_intel.fake_testing_only_20170424_151119
RUNNING TESTS:
  TESTRUNPASS_Mmpi-serial.f19_g16_rx1.A.yellowstone_intel
Starting CREATE_NEWCASE for test TESTRUNPASS_Mmpi-serial.f19_g16_rx1.A.yellowstone_intel with 1 procs
Finished CREATE_NEWCASE for test TESTRUNPASS_Mmpi-serial.f19_g16_rx1.A.yellowstone_intel in 1.889766 seconds (FAIL). [COMPLETED 1 of 1]
    Case dir: /glade/scratch/sacks/scripts_regression_test.20170424_151039/TESTRUNPASS_Mmpi-serial.f19_g16_rx1.A.yellowstone_intel.fake_testing_only_20170424_151119
    Errors were:
        WARNING: No cesm Model version found.
        Traceback (most recent call last):
          File "/glade/p/work/sacks/cime_for_comparison/scripts/create_newcase", line 213, in <module>
            _main_func(__doc__)
          File "/glade/p/work/sacks/cime_for_comparison/scripts/create_newcase", line 196, in _main_func
            input_dir=input_dir)
          File "/glade/p/work/sacks/cime_for_comparison/scripts/Tools/../../scripts/lib/CIME/case.py", line 812, in configure
            self._initialize_derived_attributes()
          File "/glade/p/work/sacks/cime_for_comparison/scripts/Tools/../../scripts/lib/CIME/case.py", line 152, in _initialize_derived_attributes
            if "aprun" in executable:
        TypeError: argument of type 'NoneType' is not iterable

At test-scheduler close, state is:
FAIL TESTRUNPASS_Mmpi-serial.f19_g16_rx1.A.yellowstone_intel (phase CREATE_NEWCASE)
    Case dir: /glade/scratch/sacks/scripts_regression_test.20170424_151039/TESTRUNPASS_Mmpi-serial.f19_g16_rx1.A.yellowstone_intel.fake_testing_only_20170424_151119
test-scheduler took 2.00474286079 seconds
ERRPUT:


======================================================================
FAIL: test_cime_case_xmlchange_append (__main__.K_TestCimeCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./scripts_regression_tests.py", line 1391, in test_cime_case_xmlchange_append
    % (SCRIPT_DIR, self._baseline_name, self._testroot, self._testroot))
  File "./scripts_regression_tests.py", line 53, in run_cmd_assert_result
    test_obj.assertEqual(stat, expected_stat, msg=msg)
AssertionError:
COMMAND: /glade/p/work/sacks/cime_for_comparison/scripts/create_test TESTRUNPASS_Mmpi-serial.f19_g16_rx1.A -t fake_testing_only_20170424_151121 --no-build --test-root /glade/scratch/sacks/scripts_regression_test.20170424_151039 --output-root /glade/scratch/sacks/scripts_regression_test.20170424_151039 --force-procs 16 --force-threads 8
FROM_DIR: /glade/p/work/sacks/cime_for_comparison/scripts/tests
SHOULD HAVE WORKED, INSTEAD GOT STAT 100
OUTPUT: Testnames: ['TESTRUNPASS_Mmpi-serial.f19_g16_rx1.A.yellowstone_intel']
Using project from .cesm_proj: P93300601
Creating test directory /glade/scratch/sacks/scripts_regression_test.20170424_151039/TESTRUNPASS_Mmpi-serial_P16x8.f19_g16_rx1.A.yellowstone_intel.fake_testing_only_20170424_151121
RUNNING TESTS:
  TESTRUNPASS_Mmpi-serial_P16x8.f19_g16_rx1.A.yellowstone_intel
Starting CREATE_NEWCASE for test TESTRUNPASS_Mmpi-serial_P16x8.f19_g16_rx1.A.yellowstone_intel with 1 procs
Finished CREATE_NEWCASE for test TESTRUNPASS_Mmpi-serial_P16x8.f19_g16_rx1.A.yellowstone_intel in 1.899996 seconds (FAIL). [COMPLETED 1 of 1]
    Case dir: /glade/scratch/sacks/scripts_regression_test.20170424_151039/TESTRUNPASS_Mmpi-serial_P16x8.f19_g16_rx1.A.yellowstone_intel.fake_testing_only_20170424_151121
    Errors were:
        WARNING: No cesm Model version found.
        Traceback (most recent call last):
          File "/glade/p/work/sacks/cime_for_comparison/scripts/create_newcase", line 213, in <module>
            _main_func(__doc__)
          File "/glade/p/work/sacks/cime_for_comparison/scripts/create_newcase", line 196, in _main_func
            input_dir=input_dir)
          File "/glade/p/work/sacks/cime_for_comparison/scripts/Tools/../../scripts/lib/CIME/case.py", line 812, in configure
            self._initialize_derived_attributes()
          File "/glade/p/work/sacks/cime_for_comparison/scripts/Tools/../../scripts/lib/CIME/case.py", line 152, in _initialize_derived_attributes
            if "aprun" in executable:
        TypeError: argument of type 'NoneType' is not iterable

At test-scheduler close, state is:
FAIL TESTRUNPASS_Mmpi-serial_P16x8.f19_g16_rx1.A.yellowstone_intel (phase CREATE_NEWCASE)
    Case dir: /glade/scratch/sacks/scripts_regression_test.20170424_151039/TESTRUNPASS_Mmpi-serial_P16x8.f19_g16_rx1.A.yellowstone_intel.fake_testing_only_20170424_151121
test-scheduler took 2.00489020348 seconds
ERRPUT:


----------------------------------------------------------------------
Ran 6 tests in 44.299s

FAILED (failures=3)
Detected failures, leaving directory: /glade/scratch/sacks/scripts_regression_test.20170424_151039
```


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/case.py b/scripts/lib/CIME/case.py
--- a/scripts/lib/CIME/case.py
+++ b/scripts/lib/CIME/case.py
@@ -149,7 +149,7 @@ def _initialize_derived_attributes(self):
             }
 
         executable = env_mach_spec.get_mpirun(self, mpi_attribs, job="case.run", exe_only=True)[0]
-        if "aprun" in executable:
+        if executable is not None and "aprun" in executable:
             self.num_nodes = get_aprun_cmd_for_case(self, "acme.exe")[1]
             self.spare_nodes = env_mach_pes.get_spare_nodes(self.num_nodes)
             self.num_nodes += self.spare_nodes
@@ -1123,7 +1123,7 @@ def get_mpirun_cmd(self, job="case.run"):
         executable, args = env_mach_specific.get_mpirun(self, mpi_attribs, job=job)
 
         # special case for aprun
-        if  "aprun" in executable:
+        if  executable is not None and "aprun" in executable:
             aprun_args, num_nodes = get_aprun_cmd_for_case(self, run_exe)
             expect(num_nodes == self.num_nodes, "Not using optimized num nodes")
             return executable + aprun_args + " " + run_misc_suffix


Test patch: