Repository: spack/spack

Issue:
openfoam: broken modules (with zsh)
Looks like something is wrong with modules from `openfoam`. If I generate module files, spack shows an error for `openfoam` package:

```
==> Warning: Quotes in command arguments can confuse scripts like configure.
  The following arguments may cause problems when executed:
      source /spack/opt/spack/linux-centos7-skylake_avx512/gcc-9.2.0/openfoam-1906-gf5sdxweeqsj2awgyc5bu2uy5grudvmf/etc/bashrc &> /dev/null && python -c "import os, json; print(json.dumps(dict(os.environ)))"
  Quotes aren't needed because spack doesn't use a shell.
  Consider removing them
==> OpenFOAM bashrc env: /spack/opt/spack/linux-centos7-skylake_avx512/gcc-9.2.0/openfoam-1906-gf5sdxweeqsj2awgyc5bu2uy5grudvmf/etc/bashrc
==> Warning: Quotes in command arguments can confuse scripts like configure.
  The following arguments may cause problems when executed:
      source /spack/opt/spack/linux-centos7-skylake_avx512/gcc-9.2.0/openfoam-1906-gf5sdxweeqsj2awgyc5bu2uy5grudvmf/etc/bashrc &> /dev/null && python -c "import os, json; print(json.dumps(dict(os.environ)))"
  Quotes aren't needed because spack doesn't use a shell.
  Consider removing them
==> OpenFOAM bashrc env: /spack/opt/spack/linux-centos7-skylake_avx512/gcc-9.2.0/openfoam-1906-gf5sdxweeqsj2awgyc5bu2uy5grudvmf/etc/bashrc
```

when loading a module, I get from `lmod`:
```
invalid command
name ""
```
or from `environment modules`:
```
Module ERROR: invalid command name ""
```

This has to do with 
```
prepend-path BASH_FUNC__module_raw() "() {  unset _mlshdbg;
...blahblahblah...
}"
```
If I manually delete it, I can at least load modules and get environment variables set correctly.


Discussion:
@olesenm do you happen to have an idea what could be wrong with openfoam modules?
@alalazo is the guru for these things.
@alalazo if/when you have a moment, could you please have a look at this module thing issue? The problem persist even with latest 0.14.2:

```
spack --stacktrace load openfoam 
lib/spack/spack/util/executable.py:152 ==> Warning: Quotes in command arguments can confuse scripts like configure.
  The following arguments may cause problems when executed:
      source /dev/null &> /dev/null && python -c "import os, json; print(json.dumps(dict(os.environ)))"
  Quotes aren't needed because spack doesn't use a shell.
  Consider removing them
lib/spack/spack/util/executable.py:152 ==> Warning: Quotes in command arguments can confuse scripts like configure.
  The following arguments may cause problems when executed:
      source /spack/opt/spack/linux-centos7-skylake_avx512/gcc-9.2.0/openfoam-1912_200403-dpmvivsqssys6kbqhzvjklroym3mjpyp/etc/bashrc &> /dev/null && python -c "import os, json; print(json.dumps(dict(os.environ)))"
  Quotes aren't needed because spack doesn't use a shell.
  Consider removing them
spack:140: no such file or directory: lib/spack/spack/user_environment.py:89
```

This is no longer related to `BASH_FUNC__module_raw` (the function is not even there).

A workaround is to use modules manually:  `module load openfoam-1912_200403-gcc-9.2.0-dpmvivs`.
@davydden @olesenm I think this is still related to `BASH_FUNC__module_raw` coming from the bashrc file that is sourced in `setup_run_environment`. I think if you add `BASH_FUNC_*` to the blacklist in that method I think it should resolve it.

I have not installed openfoam locally to test as I'm hoping someone can test my theory without having to spend the time to install openfoam, but I can try it out if you don't have an installation handy to test it out on.
Hi @becker33 

I have used `grep -i BASH` on all openfoam module files generated by spack on my installation (openfoam 1806,1812,1906,1912) and I don't see any instances at all.

if you want to test a minimal (non-functional) openfoam installation without the long compilation times, there is a backdoor that I use. It could be more elegant, but is really only ever meant as a diagnostics hack.

Change the `packages/openfoam/package.py` around line 342 to include the `common/spack-dummy-Allwmake':
```
    # General patches
    common = ['spack-Allwmake', 'README-spack', 'spack-dummy-Allwmake']
```

And later around line 372, specify that it should be used:
```
    build_script = './spack-dummy-Allwmake'
```

This builds the equivalent of "hello world" with a single application and a single library. It builds in a a few seconds and satisfies the rest of the build and installation process, which makes it perfect for testing the rest of the installation logic without waiting around (ie, good for quickly testing your hypothesis and hammering out details).

Hope that helps,

/mark
@alalazo @becker33 I was able to reproduce the `spack:140: no such file or directory: lib/spack/spack/user_environment.py:89` issue on Spack 0.14.2 as described above. But I did not achieve to generate on this version of Spack an openfoam modulefile which includes the `prepend-path BASH_FUNC__module_raw()` command.

On current Spack develop `spack load openfoam` works fine. No more warning or error message.

So from what I observe with my basic tests, there is no more issue nowadays either on modulefile generation or `spack load` sides.

What could be done however is to add `BASH_FUNC__module_raw()` to the environment variable exclusion list in `lib/spack/spack/util/environment.py`. `_module_raw()` is the shell function coming together with `module()` on Environment Modules to redirect output from stderr to stdout. I will elaborate a pull request for that.

Resolving patch:
diff --git a/lib/spack/spack/util/environment.py b/lib/spack/spack/util/environment.py
--- a/lib/spack/spack/util/environment.py
+++ b/lib/spack/spack/util/environment.py
@@ -768,16 +768,21 @@ def from_sourcing_file(
                 "PS1",
                 "PS2",
                 "ENV",
-                # Environment modules v4
+                # Environment Modules or Lmod
                 "LOADEDMODULES",
                 "_LMFILES_",
-                "BASH_FUNC_module()",
                 "MODULEPATH",
-                "MODULES_(.*)",
+                "MODULERCFILE",
+                "BASH_FUNC_ml()",
+                "BASH_FUNC_module()",
+                # Environment Modules-specific configuration
+                "MODULESHOME",
+                "BASH_FUNC__module_raw()",
+                r"MODULES_(.*)",
+                r"__MODULES_(.*)",
                 r"(\w*)_mod(quar|share)",
-                # Lmod configuration
+                # Lmod-specific configuration
                 r"LMOD_(.*)",
-                "MODULERCFILE",
             ]
         )
 


Test patch:
diff --git a/lib/spack/spack/test/data/sourceme_modules.sh b/lib/spack/spack/test/data/sourceme_modules.sh
new file mode 100644
--- /dev/null
+++ b/lib/spack/spack/test/data/sourceme_modules.sh
@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+#
+# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+_module_raw() { return 1; };
+module() { return 1; };
+ml() { return 1; };
+export -f _module_raw;
+export -f module;
+export -f ml;
+
+export MODULES_AUTO_HANDLING=1
+export __MODULES_LMCONFLICT=bar&foo
+export NEW_VAR=new
diff --git a/lib/spack/spack/test/environment_modifications.py b/lib/spack/spack/test/environment_modifications.py
--- a/lib/spack/spack/test/environment_modifications.py
+++ b/lib/spack/spack/test/environment_modifications.py
@@ -400,7 +400,7 @@ def test_sanitize_literals(env, exclude, include):
         ({"SHLVL": "1"}, ["SH.*"], [], [], ["SHLVL"]),
         # Check we can include using a regex
         ({"SHLVL": "1"}, ["SH.*"], ["SH.*"], ["SHLVL"], []),
-        # Check regex to exclude Modules v4 related vars
+        # Check regex to exclude Environment Modules related vars
         (
             {"MODULES_LMALTNAME": "1", "MODULES_LMCONFLICT": "2"},
             ["MODULES_(.*)"],
@@ -415,6 +415,13 @@ def test_sanitize_literals(env, exclude, include):
             [],
             ["A_modquar", "b_modquar", "C_modshare"],
         ),
+        (
+            {"__MODULES_LMTAG": "1", "__MODULES_LMPREREQ": "2"},
+            ["__MODULES_(.*)"],
+            [],
+            [],
+            ["__MODULES_LMTAG", "__MODULES_LMPREREQ"],
+        ),
     ],
 )
 def test_sanitize_regex(env, exclude, include, expected, deleted):
@@ -489,3 +496,19 @@ def test_exclude_lmod_variables():
     # Check that variables related to lmod are not in there
     modifications = env.group_by_name()
     assert not any(x.startswith("LMOD_") for x in modifications)
+
+
+@pytest.mark.skipif(sys.platform == "win32", reason="Not supported on Windows (yet)")
+@pytest.mark.regression("13504")
+def test_exclude_modules_variables():
+    # Construct the list of environment modifications
+    file = os.path.join(datadir, "sourceme_modules.sh")
+    env = EnvironmentModifications.from_sourcing_file(file)
+
+    # Check that variables related to modules are not in there
+    modifications = env.group_by_name()
+    assert not any(x.startswith("MODULES_") for x in modifications)
+    assert not any(x.startswith("__MODULES_") for x in modifications)
+    assert not any(x.startswith("BASH_FUNC_ml") for x in modifications)
+    assert not any(x.startswith("BASH_FUNC_module") for x in modifications)
+    assert not any(x.startswith("BASH_FUNC__module_raw") for x in modifications)