Repository: spack/spack

Issue:
tcl configuration no longer working after commit e72f87ec
### Steps to reproduce

First, set up a clean Spack environment:
```console
git clone https://github.com/spack/spack.git
cd spack
source spack/share/spack/setup-env.sh
```

Next create an environment with a package (`git` in this case) with some desired configuration:
```
spack env create test
spack env activate test
echo "spack:                              
  specs: [git]
  view: false
  concretization: together
  modules:
    enable:
    - tcl
    tcl:
      hash_length: 0
      naming_scheme: '\${COMPILERNAME}/\${COMPILERVER}/\${PACKAGE}/\${VERSION}'
      all:
        environment:
          set:
            '{name}_ROOT': '{prefix}'
" > var/spack/environments/test/spack.yaml 
```

Note the last part of the `spack.yaml` file. The intention is to have environment-module files with names like `gcc/11.2.1/<package>`, such that we have separate 'stacks' of packages using the same compiler version.

Moreover, we find it convenient to have `<package>_ROOT` variables for all the environment-modules.



### Error message

With commit `e72f87ec`, the resulting module files look like this (incorrect):

```console
$ ls share/spack/modules/linux-fedora35-zen3/
autoconf-2.69-gcc-11.2.1-3kwhasq        curl-7.81.0-gcc-11.2.1-qzfinjo    gettext-0.21-gcc-11.2.1-na5wf4d          libiconv-1.16-gcc-11.2.1-osltowj    libtool-2.4.6-gcc-11.2.1-nvvvfe6        ncurses-6.2-gcc-11.2.1-atmqel6     perl-5.34.0-gcc-11.2.1-xexayqs    xz-5.2.5-gcc-11.2.1-s7l4upo
automake-1.16.5-gcc-11.2.1-24b4ad5      diffutils-3.8-gcc-11.2.1-n4igklb  git-2.35.1-gcc-11.2.1-4kdmpue            libidn2-2.3.0-gcc-11.2.1-s6ofssk    libunistring-0.9.10-gcc-11.2.1-c5lwnzw  openssh-8.9p1-gcc-11.2.1-uohbm3i   pkgconf-1.8.0-gcc-11.2.1-2sqvvvb  zlib-1.2.11-gcc-11.2.1-uhlnocu
berkeley-db-18.1.40-gcc-11.2.1-vmxo5pc  expat-2.4.6-gcc-11.2.1-7h3fuaj    libbsd-0.11.5-gcc-11.2.1-5rfhjze         libmd-1.0.4-gcc-11.2.1-tajia74      libxml2-2.9.12-gcc-11.2.1-heehdjo       openssl-1.1.1n-gcc-11.2.1-lamvdck  readline-8.1-gcc-11.2.1-hbb5adu
bzip2-1.0.8-gcc-11.2.1-v3le5se          gdbm-1.19-gcc-11.2.1-v2ns3jx      libedit-3.1-20210216-gcc-11.2.1-2zqfxri  libsigsegv-2.13-gcc-11.2.1-5urtt6j  m4-1.4.19-gcc-11.2.1-gywxoin            pcre2-10.39-gcc-11.2.1-okwl34g     tar-1.34-gcc-11.2.1-tgt2jw2
```

And `git-2.35.1-gcc-11.2.1-4kdmpue` lacks `GIT_ROOT`.

While with `e72f87ec~1`, the environment-modules looks like expected:
```console
$ ls share/spack/modules/linux-fedora35-zen3/gcc/11.2.1/
autoconf  automake  berkeley-db  bzip2  curl  diffutils  expat  gdbm  gettext  git  libbsd  libedit  libiconv  libidn2  libmd  libsigsegv  libtool  libunistring  libxml2  m4  ncurses  openssh  openssl  pcre2  perl  pkgconf  readline  tar  xz  zlib
```

With `share/spack/modules/linux-fedora35-zen3/gcc/11.2.1/git/2.35.1` containing:
```
setenv GIT_ROOT "/scratch/spack/opt/spack/linux-fedora35-zen3/gcc-11.2.1/git-2.35.1-4kdmpuevbkiczoewdsg4zetjq7b6iuf2"
```

### Information on your system

* **Spack:** 0.17.1-880-612430859e
* **Python:** 3.10.3
* **Platform:** linux-fedora35-zen3
* **Concretizer:** clingo

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:

Resolving patch:
diff --git a/lib/spack/spack/modules/common.py b/lib/spack/spack/modules/common.py
--- a/lib/spack/spack/modules/common.py
+++ b/lib/spack/spack/modules/common.py
@@ -57,11 +57,7 @@
 #: config section for this file
 def configuration(module_set_name):
     config_path = 'modules:%s' % module_set_name
-    config = spack.config.get(config_path, {})
-    if not config and module_set_name == 'default':
-        # return old format for backward compatibility
-        return spack.config.get('modules', {})
-    return config
+    return spack.config.get(config_path, {})
 
 
 #: Valid tokens for naming scheme and env variable names
@@ -233,11 +229,6 @@ def root_path(name, module_set_name):
     # Root folders where the various module files should be written
     roots = spack.config.get('modules:%s:roots' % module_set_name, {})
 
-    # For backwards compatibility, read the old module roots for default set
-    if module_set_name == 'default':
-        roots = spack.config.merge_yaml(
-            spack.config.get('config:module_roots', {}), roots)
-
     # Merge config values into the defaults so we prefer configured values
     roots = spack.config.merge_yaml(defaults, roots)
 
@@ -698,12 +689,11 @@ def configure_options(self):
     def environment_modifications(self):
         """List of environment modifications to be processed."""
         # Modifications guessed by inspecting the spec prefix
-        std_prefix_inspections = spack.config.get(
-            'modules:prefix_inspections', {})
-        set_prefix_inspections = spack.config.get(
-            'modules:%s:prefix_inspections' % self.conf.name, {})
-        prefix_inspections = spack.config.merge_yaml(
-            std_prefix_inspections, set_prefix_inspections)
+        prefix_inspections = syaml.syaml_dict()
+        spack.config.merge_yaml(prefix_inspections, spack.config.get(
+            'modules:prefix_inspections', {}))
+        spack.config.merge_yaml(prefix_inspections, spack.config.get(
+            'modules:%s:prefix_inspections' % self.conf.name, {}))
 
         use_view = spack.config.get(
             'modules:%s:use_view' % self.conf.name, False)
@@ -939,7 +929,9 @@ def disable_modules():
     """Disable the generation of modulefiles within the context manager."""
     data = {
         'modules:': {
-            'enable': []
+            'default': {
+                'enable': []
+            }
         }
     }
     disable_scope = spack.config.InternalConfigScope('disable_modules', data=data)
diff --git a/lib/spack/spack/schema/config.py b/lib/spack/spack/schema/config.py
--- a/lib/spack/spack/schema/config.py
+++ b/lib/spack/spack/schema/config.py
@@ -56,22 +56,6 @@
                 'type': 'array',
                 'items': {'type': 'string'}
             },
-            'module_roots': {
-                'type': 'object',
-                'additionalProperties': False,
-                'properties': {
-                    'tcl': {'type': 'string'},
-                    'lmod': {'type': 'string'},
-                    'dotkit': {'type': 'string'},
-                },
-                'deprecatedProperties': {
-                    'properties': ['dotkit'],
-                    'message': 'specifying a "dotkit" module root has no '
-                               'effect [support for "dotkit" has been '
-                               'dropped in v0.13.0]',
-                    'error': False
-                },
-            },
             'source_cache': {'type': 'string'},
             'misc_cache': {'type': 'string'},
             'connect_timeout': {'type': 'integer', 'minimum': 0},
@@ -108,6 +92,12 @@
                 'items': {'type': 'string'}
             }
         },
+        'deprecatedProperties': {
+            'properties': ['module_roots'],
+            'message': 'config:module_roots has been replaced by '
+                       'modules:[module set]:roots and is ignored',
+            'error': False
+        }
     },
 }
 
diff --git a/lib/spack/spack/schema/modules.py b/lib/spack/spack/schema/modules.py
--- a/lib/spack/spack/schema/modules.py
+++ b/lib/spack/spack/schema/modules.py
@@ -8,6 +8,8 @@
 .. literalinclude:: _spack_root/lib/spack/spack/schema/modules.py
    :lines: 13-
 """
+import warnings
+
 import spack.schema.environment
 import spack.schema.projections
 
@@ -21,8 +23,8 @@
              r'defaults)(^\w[\w-]*)'
 
 #: Matches a valid name for a module set
-# Banned names are valid entries at that level in the previous schema
-set_regex = r'(?!enable|lmod|tcl|dotkit|prefix_inspections)^\w[\w-]*'
+valid_module_set_name = r'^(?!arch_folder$|lmod$|roots$|enable$|prefix_inspections$|'\
+                        r'tcl$|use_view$)\w[\w-]*$'
 
 #: Matches an anonymous spec, i.e. a spec without a root name
 anonymous_spec_regex = r'^[\^@%+~]'
@@ -117,24 +119,12 @@
 }
 
 
-#: The "real" module properties -- the actual configuration parameters.
-#: They are separate from ``properties`` because they can appear both
-#: at the top level of a Spack ``modules:`` config (old, deprecated format),
-#: and within a named module set (new format with multiple module sets).
 module_config_properties = {
     'use_view': {'anyOf': [
         {'type': 'string'},
         {'type': 'boolean'}
     ]},
     'arch_folder': {'type': 'boolean'},
-    'prefix_inspections': {
-        'type': 'object',
-        'additionalProperties': False,
-        'patternProperties': {
-            # prefix-relative path to be inspected for existence
-            r'^[\w-]*': array_of_strings
-        }
-    },
     'roots': {
         'type': 'object',
         'properties': {
@@ -147,15 +137,8 @@
         'default': [],
         'items': {
             'type': 'string',
-            'enum': ['tcl', 'dotkit', 'lmod']
-        },
-        'deprecatedProperties': {
-            'properties': ['dotkit'],
-            'message': 'cannot enable "dotkit" in modules.yaml '
-            '[support for "dotkit" has been dropped '
-            'in v0.13.0]',
-            'error': False
-        },
+            'enum': ['tcl', 'lmod']
+        }
     },
     'lmod': {
         'allOf': [
@@ -178,40 +161,54 @@
             {}  # Specific tcl extensions
         ]
     },
-    'dotkit': {
-        'allOf': [
-            # Base configuration
-            module_type_configuration,
-            {}  # Specific dotkit extensions
-        ]
+    'prefix_inspections': {
+        'type': 'object',
+        'additionalProperties': False,
+        'patternProperties': {
+            # prefix-relative path to be inspected for existence
+            r'^[\w-]*': array_of_strings
+        }
     },
 }
 
 
+def deprecation_msg_default_module_set(instance, props):
+    return (
+        'Top-level properties "{0}" in module config are ignored as of Spack v0.18. '
+        'They should be set on the "default" module set. Run\n\n'
+        '\t$ spack config update modules\n\n'
+        'to update the file to the new format'.format('", "'.join(instance))
+    )
+
+
 # Properties for inclusion into other schemas (requires definitions)
 properties = {
     'modules': {
         'type': 'object',
+        'additionalProperties': False,
+        'properties': {
+            'prefix_inspections': {
+                'type': 'object',
+                'additionalProperties': False,
+                'patternProperties': {
+                    # prefix-relative path to be inspected for existence
+                    r'^[\w-]*': array_of_strings
+                }
+            },
+        },
         'patternProperties': {
-            set_regex: {
+            valid_module_set_name: {
                 'type': 'object',
                 'default': {},
                 'additionalProperties': False,
-                'properties': module_config_properties,
-                'deprecatedProperties': {
-                    'properties': ['dotkit'],
-                    'message': 'the "dotkit" section in modules.yaml has no effect'
-                    ' [support for "dotkit" has been dropped in v0.13.0]',
-                    'error': False
-                }
+                'properties': module_config_properties
             },
+            # Deprecated top-level keys (ignored in 0.18 with a warning)
+            '^(arch_folder|lmod|roots|enable|tcl|use_view)$': {}
         },
-        # Available here for backwards compatibility
-        'properties': module_config_properties,
         'deprecatedProperties': {
-            'properties': ['dotkit'],
-            'message': 'the "dotkit" section in modules.yaml has no effect'
-            ' [support for "dotkit" has been dropped in v0.13.0]',
+            'properties': ['arch_folder', 'lmod', 'roots', 'enable', 'tcl', 'use_view'],
+            'message': deprecation_msg_default_module_set,
             'error': False
         }
     }
@@ -225,3 +222,39 @@
     'additionalProperties': False,
     'properties': properties,
 }
+
+
+def update(data):
+    """Update the data in place to remove deprecated properties.
+
+    Args:
+        data (dict): dictionary to be updated
+
+    Returns:
+        True if data was changed, False otherwise
+    """
+    changed = False
+
+    deprecated_top_level_keys = ('arch_folder', 'lmod', 'roots', 'enable',
+                                 'tcl', 'use_view')
+
+    # Don't update when we already have a default module set
+    if 'default' in data:
+        if any(key in data for key in deprecated_top_level_keys):
+            warnings.warn('Did not move top-level module properties into "default" '
+                          'module set, because the "default" module set is already '
+                          'defined')
+        return changed
+
+    default = {}
+
+    # Move deprecated top-level keys under "default" module set.
+    for key in deprecated_top_level_keys:
+        if key in data:
+            default[key] = data.pop(key)
+
+    if default:
+        changed = True
+        data['default'] = default
+
+    return changed


Test patch:
diff --git a/lib/spack/spack/test/bootstrap.py b/lib/spack/spack/test/bootstrap.py
--- a/lib/spack/spack/test/bootstrap.py
+++ b/lib/spack/spack/test/bootstrap.py
@@ -73,15 +73,15 @@ def test_bootstrap_deactivates_environments(active_mock_environment):
 @pytest.mark.regression('25805')
 def test_bootstrap_disables_modulefile_generation(mutable_config):
     # Be sure to enable both lmod and tcl in modules.yaml
-    spack.config.set('modules:enable', ['tcl', 'lmod'])
+    spack.config.set('modules:default:enable', ['tcl', 'lmod'])
 
-    assert 'tcl' in spack.config.get('modules:enable')
-    assert 'lmod' in spack.config.get('modules:enable')
+    assert 'tcl' in spack.config.get('modules:default:enable')
+    assert 'lmod' in spack.config.get('modules:default:enable')
     with spack.bootstrap.ensure_bootstrap_configuration():
-        assert 'tcl' not in spack.config.get('modules:enable')
-        assert 'lmod' not in spack.config.get('modules:enable')
-    assert 'tcl' in spack.config.get('modules:enable')
-    assert 'lmod' in spack.config.get('modules:enable')
+        assert 'tcl' not in spack.config.get('modules:default:enable')
+        assert 'lmod' not in spack.config.get('modules:default:enable')
+    assert 'tcl' in spack.config.get('modules:default:enable')
+    assert 'lmod' in spack.config.get('modules:default:enable')
 
 
 @pytest.mark.regression('25992')
diff --git a/lib/spack/spack/test/config.py b/lib/spack/spack/test/config.py
--- a/lib/spack/spack/test/config.py
+++ b/lib/spack/spack/test/config.py
@@ -561,20 +561,6 @@ def test_read_config_override_all(mock_low_high_config, write_config_file):
     }
 
 
-@pytest.mark.regression('23663')
-def test_read_with_default(mock_low_high_config):
-    # this very synthetic example ensures that config.get(path, default)
-    # returns default if any element of path doesn't exist, regardless
-    # of the type of default.
-    spack.config.set('modules', {'enable': []})
-
-    default_conf = spack.config.get('modules:default', 'default')
-    assert default_conf == 'default'
-
-    default_enable = spack.config.get('modules:default:enable', [])
-    assert default_enable == []
-
-
 def test_read_config_override_key(mock_low_high_config, write_config_file):
     write_config_file('config', config_low, 'low')
     write_config_file('config', config_override_key, 'high')
@@ -1100,22 +1086,6 @@ def test_bad_compilers_yaml(tmpdir):
 """)
 
 
-@pytest.mark.regression('13045')
-def test_dotkit_in_config_does_not_raise(
-        mock_low_high_config, write_config_file, capsys
-):
-    write_config_file('config',
-                      {'config': {'module_roots': {'dotkit': '/some/path'}}},
-                      'high')
-    spack.main.print_setup_info('sh')
-    captured = capsys.readouterr()
-
-    # Check that we set the variables we expect and that
-    # we throw a a deprecation warning without raising
-    assert '_sp_sys_type' in captured[0]  # stdout
-    assert 'Warning' in captured[1]  # stderr
-
-
 def test_internal_config_section_override(mock_low_high_config,
                                           write_config_file):
     write_config_file('config', config_merge_list, 'low')
diff --git a/lib/spack/spack/test/modules/lmod.py b/lib/spack/spack/test/modules/lmod.py
--- a/lib/spack/spack/test/modules/lmod.py
+++ b/lib/spack/spack/test/modules/lmod.py
@@ -313,23 +313,6 @@ def test_projections_all(self, factory, module_configuration):
         projection = writer.spec.format(writer.conf.projections['all'])
         assert projection in writer.layout.use_name
 
-    def test_config_backwards_compat(self, mutable_config):
-        settings = {
-            'enable': ['lmod'],
-            'lmod': {
-                'core_compilers': ['%gcc@0.0.0']
-            }
-        }
-
-        spack.config.set('modules:default', settings)
-        new_format = spack.modules.lmod.configuration('default')
-
-        spack.config.set('modules', settings)
-        old_format = spack.modules.lmod.configuration('default')
-
-        assert old_format == new_format
-        assert old_format == settings['lmod']
-
     def test_modules_relative_to_view(
         self, tmpdir, modulefile_content, module_configuration, install_mockery,
         mock_fetch
diff --git a/lib/spack/spack/test/modules/tcl.py b/lib/spack/spack/test/modules/tcl.py
--- a/lib/spack/spack/test/modules/tcl.py
+++ b/lib/spack/spack/test/modules/tcl.py
@@ -391,25 +391,6 @@ def test_autoload_with_constraints(
         content = modulefile_content('mpileaks ^mpich')
         assert len([x for x in content if 'is-loaded' in x]) == 0
 
-    def test_config_backwards_compat(self, mutable_config):
-        settings = {
-            'enable': ['tcl'],
-            'tcl': {
-                'all': {
-                    'conflict': ['{name}']
-                }
-            }
-        }
-
-        spack.config.set('modules:default', settings)
-        new_format = spack.modules.tcl.configuration('default')
-
-        spack.config.set('modules', settings)
-        old_format = spack.modules.tcl.configuration('default')
-
-        assert old_format == new_format
-        assert old_format == settings['tcl']
-
     def test_modules_no_arch(self, factory, module_configuration):
         module_configuration('no_arch')
         module, spec = factory(mpileaks_spec_string)