Repository: ESMCI/cime

Issue:
CIMEROOT environment variable used too aggressively
If you have CIMEROOT defined in your environment (or, presumably, in your .cime_config), it appears that any scripts you run from cime load at least some things from the directory referenced from CIMEROOT, rather than the current directory.

I tested this by setting:

``` bash
export CIMEROOT=/path/does/not/exist
```

and then trying to run create_test from cime/scripts. This gave:

```
Traceback (most recent call last):
  File "./create_test", line 11, in <module>
    from Tools.standard_script_setup import *
  File "/glade/p/work/sacks/cime/scripts/Tools/standard_script_setup.py", line 15, in <module>
    import CIME.utils
ImportError: No module named CIME.utils
```

(similar things happen for create_newcase, etc.)

I'm not really concerned about this exact case (where CIMEROOT points to a non-existent directory). I was more using this as a measure of whether standard scripts like create_newcase and create_test try to load anything out of CIMEROOT - and apparently they do.

It's unclear to me, then, what comes from CIMEROOT and what comes from the current directory. 

This could be very confusing and cause incorrect results in situations like this: Someone has CIMEROOT set to point to some cime clone, and they also have a CESM checkout with a different version of cime. They run create_newcase from their CESM checkout. Would this actually be using some / all things from CIMEROOT rather than from the current directory? This, in my opinion, would be a Bad Thing.

I forget the reason why CIMEROOT was introduced originally. But I'm having trouble seeing any reason why you would ever want to run a script out of cime directory A, and have it use pieces from cime directory B. It feels like this creates potential for significant confusion and unexpected behavior.



Discussion:
PR #655 should address this issue.   Please consider reviewing it now.   

I don't think that PR #655 covers the larger issue. I think we should all agree on the proper use of CIMEROOT (if any) and make sure all the code enforces that use.
Going back to the origin of CIME, I recall that the idea was for CIME to be a unified, standalone tool set. A logical consequence is that any use of any CIME script must use the consistent version from whence it came. The function, get_python_libs_location_within_cime, seems to fulfill this purpose. What independent role does CIMEROOT play? Why is it ever needed or consulted?
What I see as a reasonable split is having CIME not be in a CESM or ACME directory tree but be able to operate on that tree. Thus, I can see create_newcase or create_test using the CESMROOT environment variable to make that link.
Is there ever a reason for a CIMEROOT environment variable? If so, please document it here (and in the CIME manual) so we can ensure consistent usage.

The only time I've needed it was to run some of the few tools that are copied/generated (not symlinked) in the case directory. 

It's like that, if you're in a case directory, there's another way to get the CIMEROOT without have to use the environment. I'll look into this.

I think that the submitted jobs : template.\* need to have it in the environment, so be sure and test changes with a batch system :-)

I would argue that this is an error as the correct CIME location is known when these files are processed. I think this also applies to config_batch.xml with lines such as:
`<arg flag="--env" name='"CIMEROOT=$CIMEROOT"'/>`

I think that you are correct - this use in config_batch.xml is no longer needed.


Resolving patch:
diff --git a/scripts/Tools/standard_script_setup.py b/scripts/Tools/standard_script_setup.py
--- a/scripts/Tools/standard_script_setup.py
+++ b/scripts/Tools/standard_script_setup.py
@@ -5,13 +5,13 @@
 
 import sys, os
 import __main__ as main
-_CIMEROOT = os.environ.get("CIMEROOT")
-if(_CIMEROOT is None):
-    _CIMEROOT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..","..")
-    os.environ["CIMEROOT"] = _CIMEROOT
+_CIMEROOT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..","..")
 _LIB_DIR = os.path.join(_CIMEROOT, "utils", "python")
 sys.path.append(_LIB_DIR)
 
+# Important: Allows external tools to link up with CIME
+os.environ["CIMEROOT"] = _CIMEROOT
+
 import CIME.utils
 CIME.utils.check_minimum_python_version(2, 7)
 CIME.utils.stop_buffering_output()
diff --git a/utils/python/CIME/case_submit.py b/utils/python/CIME/case_submit.py
--- a/utils/python/CIME/case_submit.py
+++ b/utils/python/CIME/case_submit.py
@@ -35,9 +35,6 @@ def submit(case, job=None, resubmit=False, no_batch=False):
                 with open(".original_host","w") as fd:
                     fd.write( socket.gethostname())
 
-    cimeroot = case.get_value("CIMEROOT")
-    os.environ["CIMEROOT"] = cimeroot
-
     # if case.submit is called with the no_batch flag then we assume that this
     # flag will stay in effect for the duration of the RESUBMITs
     env_batch = case.get_env("batch")
diff --git a/utils/python/CIME/utils.py b/utils/python/CIME/utils.py
--- a/utils/python/CIME/utils.py
+++ b/utils/python/CIME/utils.py
@@ -33,7 +33,6 @@ def _read_cime_config_file():
     """
     READ the config file in ~/.cime, this file may contain
     [main]
-    CIMEROOT=/path/to/cime
     CIME_MODEL=acme,cesm
     PROJECT=someprojectnumber
     """
@@ -78,22 +77,14 @@ def get_cime_root(case=None):
     >>> os.path.isdir(os.path.join(get_cime_root(), get_scripts_location_within_cime()))
     True
     """
-    cime_config = get_cime_config()
-    if(cime_config.has_option('main','CIMEROOT')):
-        cimeroot = cime_config.get('main','CIMEROOT')
-    else:
-        if "CIMEROOT" in os.environ:
-            cimeroot = os.environ["CIMEROOT"]
-        else:
-            script_absdir = os.path.abspath(os.path.join(os.path.dirname(__file__),".."))
-            assert script_absdir.endswith(get_python_libs_location_within_cime()), script_absdir
-            cimeroot = os.path.abspath(os.path.join(script_absdir,"..",".."))
-        cime_config.set('main','CIMEROOT',cimeroot)
+    script_absdir = os.path.abspath(os.path.join(os.path.dirname(__file__),".."))
+    assert script_absdir.endswith(get_python_libs_location_within_cime()), script_absdir
+    cimeroot = os.path.abspath(os.path.join(script_absdir,"..",".."))
 
     if case is not None:
         case_cimeroot = os.path.abspath(case.get_value("CIMEROOT"))
         cimeroot = os.path.abspath(cimeroot)
-        expect(cimeroot == case_cimeroot, "Inconsistant CIMEROOT variable: case %s environment %s"%(case_cimeroot, cimeroot))
+        expect(cimeroot == case_cimeroot, "Inconsistent CIMEROOT variable: case -> '%s', file location -> '%s'" % (case_cimeroot, cimeroot))
 
     logger.debug( "CIMEROOT is " + cimeroot)
     return cimeroot


Test patch:
diff --git a/tools/system_tests/configure.test/configtest.pl b/tools/system_tests/configure.test/configtest.pl
deleted file mode 100755
--- a/tools/system_tests/configure.test/configtest.pl
+++ /dev/null
@@ -1,194 +0,0 @@
-#!/usr/bin/env perl
-use strict;
-use warnings;
-use XML::LibXML;
-use Getopt::Long;
-use File::Path;
-use File::Basename qw(basename dirname);
-use Cwd;
-
-#-----------------------------------------------------------------------------------------------
-if ($#ARGV == -1) {
-    usage();
-}
-
-#-----------------------------------------------------------------------------------------------
-
-sub usage {
-    die <<EOF;
-SYNOPSIS
-     configtest.pl [options]
-OPTIONS
-     User supplied values are denoted in angle brackets (<>).  Any value that contains
-     white-space must be quoted.  Long option names may be supplied with either single
-     or double leading dashes.  A consequence of this is that single letter options may
-     NOT be bundled.
-
-     -cimeroot            Specify the toplevel cime directory.
-                          default: use CIMEROOT environment variable
-     -mach_dir <path>     Specify the locations of the Machines directory (optional).
-     -model    <name>     Specify the cime model on which to conduct tests.
-                          default: cesm
-     -baselineroot       Root directory to read or write baselines to.
-     -compare             compare against baselines in this directory (under baselineroot).
-     -generate             write new baselines to this directory (under baselineroot).
-     -help [or -h]        Print usage to STDOUT (optional).
-
-EXAMPLES
-
-  ./configtest.pl -cimeroot /path/to/cime -compare cime3.0.1 -baselineroot /path/to/baselines/
-
-EOF
-}
-my ($model, $baselineroot, $generate, $compare, $cimeroot, $machdir, $verbose, $silent, $help);
-my $logger;
-my $loglevel = "INFO";
-my $output_dir = getcwd();
-my @time = localtime();
-$output_dir .= "/$time[5]$time[4]$time[3]$time[2]$time[1]";
-sub options{
-    GetOptions(
-	"cimeroot=s"                => \$cimeroot,
-        "baselineroot=s"          => \$baselineroot,
-	"model=s"                   => \$model,
-	"h|help"                    => \$help,
-	"machdir=s"                => \$machdir,
-	"loglevel=s"               => \$loglevel,
-	"generate=s"              => \$generate,
-	"compare=s"               => \$compare,
-        "output_dir=s"           => \$output_dir,
-	)  or usage();
-
-
-    if(! defined $cimeroot) {
-	$cimeroot = abs_path($ENV{CIMEROOT}) if(defined $ENV{CIMEROOT});
-    }
-    (-d "$cimeroot")  or  die "Cannot find cimeroot directory \"$cimeroot\" " ;
-
-    my @dirs = ("$cimeroot/utils/perl5lib" );
-
-    unshift @INC, @dirs;
-    require Log::Log4perl;
-    require File::Copy::Recursive;
-    require File::DirCompare;
-
-    my $level = Log::Log4perl::Level::to_priority($loglevel);
-    Log::Log4perl->easy_init({level=>$level,
-			  layout=>'%m%n'});
-
-    $logger = Log::Log4perl::get_logger();
-# Give usage message.
-    usage() if $help;
-
-
-# Check for unparsed argumentss
-    if (@ARGV) {
-	$logger->error( "ERROR: unrecognized arguments: @ARGV");
-	usage();
-    }
-
-    if (! defined $model){
-	$model = 'cesm';
-    }
-
-    if(! defined $machdir){
-	$machdir  = "$cimeroot/cime_config/${model}/machines";
-    }
-    if(!defined $baselineroot){
-	$logger->logdie("baselineroot argument must be provided");
-    }
-    if(!defined $generate && !defined $compare){
-	$logger->logdie( "At least one of the arguments generate and compare must be provided");
-    }
-    if(defined $compare){
-	if(-d "$baselineroot/$compare"){
-	    $logger->info( "Compare to baseline directory $baselineroot/$compare");
-	}else{
-	    $logger->logdie ("Could not find baseline directory $compare");
-	}
-    }
-    if(defined $generate){
-	if(-d "$baselineroot/$generate"){
-	    $logger->logdie ("Directory $baselineroot/$generate already exists.");
-	}
-    }
-    if(defined $output_dir){
-	if(-d $output_dir){
-	    $logger->warn("Output directory $output_dir already exists");
-	}else{
-
-	}
-    }
-}
-options();
-#-----------------------------------------------------------------------------------------------
-# Make sure we can find required perl modules and configuration files.
-# Look for them in the directory that contains the configure script.
-
-# Machines definition file.
-my $machine_file = 'config_machines.xml';
-(-f "$machdir/$machine_file")  or
-    $logger->logdie("Cannot find machine parameters file $machine_file in directory $machdir ");
-
-
-my $xml = XML::LibXML->new( )->parse_file("$machdir/$machine_file");
-
-
-foreach my $node ($xml->findnodes(".//machine")){
-    my $mach = $node->getAttribute("MACH");
-    next if ($mach eq "userdefined");
-    my @child = $node->findnodes(".//COMPILERS");
-    my @compilers = split(',',$child[0]->textContent());
-
-    foreach my $compiler (@compilers){
-	foreach my $format (qw(make cmake)){
-
-	    my $test_dir = "$output_dir/$mach/$compiler/$format";
-	    unless(-d $test_dir){
-		mkpath $test_dir or $logger->logdie("Could not create directory $test_dir");
-	    }
-	    qx( $cimeroot/tools/configure -cimeroot $cimeroot -mach $mach -compiler $compiler -output_dir $test_dir -output_format $format -loglevel $loglevel);
-	}
-    }
-}
-
-if(defined $generate){
-     File::Copy::Recursive::dircopy($output_dir, "$baselineroot/$generate") or $logger->logdie("Could not copy $output_dir to $baselineroot/$generate");
-}
-
-
-if(defined $compare){
-    my $result = "PASS";
-    my $filediff =0;
-    my $nobase=0;
-    my $notest = 0;
-
-
-    $logger->info("\n\nComparing $output_dir to $baselineroot/$compare");
-    File::DirCompare->compare("$output_dir","$baselineroot/$compare",
-		     sub {
-			 my($a, $b) = @_;
-			 if( !$b){
-			     my $base = basename($a);
-			     my $dir = dirname($a);
-			     $logger->warn("  File $base only exists in $dir");
-			     $notest++;
-			 }elsif( !$a){
-			     my $base = basename($b);
-			     my $dir = dirname($b);
-			     $logger->warn("  File $base only exists in $dir");
-			     $nobase++;
-			 }else{
-			     $logger->error("  File contents for $a and $b differ.\n");
-			     $filediff++;
-			 }
-		     });
-
-    if($filediff >0){
-	print "CONFIGTEST RESULT: FAIL $notest $nobase $filediff\n";
-    }else{
-	print "CONFIGTEST RESULT: PASS\n";
-    }
-
-
-}
diff --git a/utils/python/CIME/test_scheduler.py b/utils/python/CIME/test_scheduler.py
--- a/utils/python/CIME/test_scheduler.py
+++ b/utils/python/CIME/test_scheduler.py
@@ -52,9 +52,6 @@ def __init__(self, test_names, test_data=None,
         self._queue       = queue
         self._test_data   = {} if test_data is None else test_data # Format:  {test_name -> {data_name -> data}}
 
-        # needed for perl interface
-        os.environ["CIMEROOT"] = self._cime_root
-
         self._machobj = Machines(machine=machine_name)
 
         self._no_setup = no_setup