Repository: spack/spack

Issue:
[BUG] Modules fail on old Linux (SLES 11)
 @alalazo @tgamblin @adamjstewart @mathstuf`

I'm running on an old SuSE system with obviously out-of-date software (eg: Python 2.6):

```
$ uname -a
Linux discover16 3.0.101-0.47.79-default #1 SMP Fri Apr 22 06:30:00 UTC 2016 (9d3a4e8) x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/SuSE-release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 3
```

A `module load` command of Spack-generate modules produces the following error:

```
$ module load linux-SuSE11-x86_64/m4-1.4.17-gcc-5.3.0-lheed2t
cmdPath.c(156):ERROR:11: Usage is 'prepend-pathpath-variable directory'
linux-SuSE11-x86_64/m4-1.4.17-gcc-5.3.0-lheed2t(12):ERROR:102: Tcl command execution failed: prepend-path --delim ":" PATH "/gpfsm/dnb53/rpfische/spack2/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/m4-1.4.17-lheed2tiee3dqj2e73xfqr46jgzhefys/bin"
```

This didn't used to happen.  What version of `module` and `tcl` am I running?

```
$ module --version
3.1.6
$ echo 'puts $tcversion;exit 0' | tclsh
8.5
```

I noticed that there has been a change in the format of the module files that is causing this problem.  For example, in the `binutils` module file generated by the old Spack (it loaded just fine), I have:

```
prepend-path PATH "/gpfsm/dnb53/rpfische/spack/opt/spack/linux-x86_64/gcc-5.3.0/binutils-2.26-jt3fjwoi4s5yvt7kf5jw3jzozqzcd4yn/bin"
```

The module file generated by the new Spack (which doesn't work) looks like:

```
prepend-path --delim ":" PATH "/gpfsm/dnb53/rpfische/spack2/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/binutils-2.25-6w5d2t454o53bxixyhx2ajm4wsjgb4xv/bin"
```

Any ideas?

spack modulefiles prepend-path --delim 
My system has Modules Tcl 3.1.6 and does not like  prepend-path --delim syntax 
Would it be possible to make these formats configurable in modules .yaml?

Does this problem affect someone else?   I got it by spack load <any newly installed pkg>



Discussion:
@trws The bug seems to be caused by the following commit; can you take a look at it please?

```
commit b5979b13e3f9bcf3e151b494bb2a557aaabedeff
Author: Tom Scogland <scogland1@llnl.gov>
Date:   Sat May 14 16:03:42 2016 -0700

    complete lua rework, also module path fix

    This is a complete rework of the lua package, it also allows the
    environment modification classes to handle paths that are not separated
    by colons, and uses the support for same in TCL modules as well.

    The biggest difference is the handling for lua extension packages, which
    now have their paths set correctly by the lua parent package, and have
    access to both lua and luarocks as installation tools. See the luaposix
    package for what should be required for most lua packages after this.
```

This is a duplicate of #1003, and @luigi-calori has a PR that works around it in #1358.  It's caused by using a version of TCL modules that's at least a decade old being combined with the fact that not all environment path components are separated with colons.

According to their website, Modules TCL added `--delim` in Modules 3.2.3 in 2006.  What distro are you using? Even RedHat EL/CentOS 5 is using 3.2.10 if its up to date.

I'm not averse to having an option for whether that should be supported or not I suppose, but it will limit the functionality of the lua package to not be able to select the correct delimiter for its path variables.

The problem is that our sysadms have used this old module installation as a default for all users.
So the default module interpreter for all users is this old one.
I have almost ready a PR  for let the current behaviour be customizable by adding a  delim_format definition in [modules.yaml](https://github.com/RemoteConnectionManager/spack/commit/786d1df0f1e1e20bfc8404455364a6487932e479) 

Sorry ...  pressed wrong button....
I would like to increase the ability to customize the modules in order for me to be able to comply to our internal policy of module development. 
I'm not yet  fully confident on config syntax but the idea is to let the  user override in modules.yaml the templates that are currently fixed in modules.py 

No worries.  I'm all for having it be more configurable as long as we can express the values we need, I'm sorry the --delim option caused you issues.

@luigi-calori Why not `spack install lmod`  and then use that ?

Eh... I have my trouble in trying to show spack is ready to test and to substitute our legacy way of building sw.
I would like to rely on the same module system, this way If I build packages with spack, then I could let other users test them by just doing
module use <spack root>/share/spack/modules/$(spack arch)

They even do not need to be aware modules have been built with spack 

I do not know lmod but I think it will need to be activated and not sure if it can easily interoperate with classical env modules.
I would really like spack adoption to be as less intrusive as possible, especially to users.
I also think that being able to override module templates could be of general utility: for example to add code to log module usage to all modules ( if not already possible)

> I do not know lmod but I think it will need to be activated and not sure if it can easily interoperate with classical env modules

It's completely backward compatible with tcl module files, see lmod docs. 

> I would like to rely on the same module system, this way If I build packages with spack, then I could let other users test them by just doing
> module use /share/spack/modules/$(spack arch)

What we have done here to deal with the same kind of issue is to have a short shell script to source lmod init file and add `/share/spack/modules/$(spack arch)` to the `MODULEPATH`. The same script can restore the login setting if a flag is passed.

Well, I understand lmod would be the way to go in the long run, but  
I would prefer to not change also the module system and rely on the old one.
As spack allows the production of tcl module code, I thought lmod was not strictly needed.
The --delim was added very recently so I tried to ad a PR that will allow to configure it in order to disable.
Do you think the idea of letting users override module template is dangerous or unconvenient?

> Do you think the idea of letting users override module template is dangerous or unconvenient?

No, I just don't see how much useful would this be in `develop`. Per @trws comment : 

> According to their website, Modules TCL added --delim in Modules 3.2.3 in 2006. What distro are you using? Even RedHat EL/CentOS 5 is using 3.2.10 if its up to date.

we would need this change to support version of environment modules that pre-date 2006. Just to give you an idea of how old this is, think of `gcc-3.4.6`... 

MM.. I do not know the reason, but on all our clusters we still have module 3.1.6. 
I agree that this is really old and I' ve requested to upgrade.
Nevertheless,  allowing users override  modulefile templates  string defaults  could be useful in other cases. But maybe there are reason for not going in such direction.
If no one else is interested, I' ll keep the PR live until I manage to upgrade my module system.
Maybe you could evaluate the idea of making module code generation more configurable by letting user override  the strings used for template in module.yaml


Resolving patch:
diff --git a/lib/spack/spack/modules.py b/lib/spack/spack/modules.py
--- a/lib/spack/spack/modules.py
+++ b/lib/spack/spack/modules.py
@@ -530,13 +530,6 @@ def prerequisite(self, spec):
 class TclModule(EnvModule):
     name = 'tcl'
     path = join_path(spack.share_path, "modules")
-    environment_modifications_formats = {
-        PrependPath: 'prepend-path --delim "{separator}" {name} \"{value}\"\n',
-        AppendPath: 'append-path   --delim "{separator}" {name} \"{value}\"\n',
-        RemovePath: 'remove-path   --delim "{separator}" {name} \"{value}\"\n',
-        SetEnv: 'setenv {name} \"{value}\"\n',
-        UnsetEnv: 'unsetenv {name}\n'
-    }
 
     autoload_format = ('if ![ is-loaded {module_file} ] {{\n'
                        '    puts stderr "Autoloading {module_file}"\n'
@@ -556,11 +549,13 @@ def file_name(self):
     def header(self):
         timestamp = datetime.datetime.now()
         # TCL Modulefile header
-        header = '#%Module1.0\n'
-        header += '## Module file created by spack (https://github.com/LLNL/spack) on %s\n' % timestamp
-        header += '##\n'
-        header += '## %s\n' % self.spec.short_spec
-        header += '##\n'
+        header = """\
+#%%Module1.0
+## Module file created by spack (https://github.com/LLNL/spack) on %s
+##
+## %s
+##
+""" % (timestamp, self.spec.short_spec)
 
         # TODO : category ?
         # Short description
@@ -575,6 +570,44 @@ def header(self):
             header += '}\n\n'
         return header
 
+    def process_environment_command(self, env):
+        environment_modifications_formats_colon = {
+            PrependPath: 'prepend-path {name} \"{value}\"\n',
+            AppendPath: 'append-path   {name} \"{value}\"\n',
+            RemovePath: 'remove-path   {name} \"{value}\"\n',
+            SetEnv: 'setenv {name} \"{value}\"\n',
+            UnsetEnv: 'unsetenv {name}\n'
+        }
+        environment_modifications_formats_general = {
+            PrependPath:
+            'prepend-path --delim "{separator}" {name} \"{value}\"\n',
+            AppendPath:
+            'append-path   --delim "{separator}" {name} \"{value}\"\n',
+            RemovePath:
+            'remove-path   --delim "{separator}" {name} \"{value}\"\n',
+            SetEnv: 'setenv {name} \"{value}\"\n',
+            UnsetEnv: 'unsetenv {name}\n'
+        }
+        for command in env:
+            # Token expansion from configuration file
+            name = command.args.get('name', '').format(**self.upper_tokens)
+            value = str(command.args.get('value', '')).format(**self.tokens)
+            command.update_args(name=name, value=value)
+            # Format the line int the module file
+            try:
+                if command.args.get('separator', ':') == ':':
+                    yield environment_modifications_formats_colon[type(
+                        command)].format(**command.args)
+                else:
+                    yield environment_modifications_formats_general[type(
+                        command)].format(**command.args)
+            except KeyError:
+                message = ('Cannot handle command of type {command}: '
+                           'skipping request')
+                details = '{context} at {filename}:{lineno}'
+                tty.warn(message.format(command=type(command)))
+                tty.warn(details.format(**command.args))
+
     def module_specific_content(self, configuration):
         naming_tokens = self.tokens
         # Conflict


Test patch: