Repository: spack/spack

Issue:
PYTHONPATHs not set for python components after releases/v0.20
### Discussed in https://github.com/spack/spack/discussions/42433

<div type='discussions-op-text'>

<sup>Originally posted by **jgalarowicz** February  1, 2024</sup>
I tried to use spack releases instead of the development branch to build survey in hopes of figuring out why we don't get the PYTHONPATH statements in the survey module file. 

I'm asking spack not to load the module files for all the components by using autoload: none feature.

    tcl:
      all:
        autoload: none

    # Default configurations if lmod is enabled
    lmod:
      all:
        autoload: none

When building survey with spack releases/v0.20, spack puts all the PYTHONPATH statements into the module file for survey.   Releases after v0.20 do not put the PYTHONPATH statements in.

I would like to know how we can get this functionality back.
With releases/v0.21 and above we only get the PYTHONPATH for survey, but not the components.
When we try to execute survey we get python module not found error messages.
This is from releases/v0.20 generated spack survey module file:

    prepend-path --delim {:} PATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/survey-1.0.9-752pn6r4sbegdgxjmllgvcn5jgw2xhny/bin}
    prepend-path --delim {:} MANPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/survey-1.0.9-752pn6r4sbegdgxjmllgvcn5jgw2xhny/share/man}
    prepend-path --delim {:} CMAKE_PREFIX_PATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/survey-1.0.9-752pn6r4sbegdgxjmllgvcn5jgw2xhny/.}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/survey-1.0.9-752pn6r4sbegdgxjmllgvcn5jgw2xhny/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-filelock-3.12.0-c5jsuadbwnrbziolwzxuqe7wxumgqbt5/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-gitpython-3.1.27-fm7ihfpoxxs6agnxi763tbv2zqxaadm3/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-gitdb-4.0.9-boc5ep6bdwbztaotophctipa3o4prcqg/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-smmap-5.0.0-q2ndbaghziznfcxgkpuxex5tm7u5wp3k/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-humanize-4.4.0-sjltgwocdkdff7jx7o44wssrrp66ozhs/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-setuptools-63.4.3-bv5g53m2o2aeknmllawhhdiac6ljq3jp/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-importlib-resources-5.12.0-z5b4ljqcu4fb4plgaaf7fbk5c6sdkiln/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-jinja2-3.1.2-yecqrma22ebxn3velzhsfouuzwqv4akn/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-markupsafe-2.1.1-cnna6ntbcddumvillznr7oifx27kup7k/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-matplotlib-3.7.1-nrlrcxi3yjbzi2fczfwfkl4l7xe7mfuj/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-contourpy-1.0.5-kxzyyx32bt3pqcrxay33y47ljoagzqyj/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-numpy-1.24.3-xgkos7qlv34h55ndnyoh2cd2pkcebiii/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-cycler-0.11.0-zlwfn2l4m66dnexehbvf4hgdaf6jndsr/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-fonttools-4.37.3-pygiqxp6qlp6ydvtcaawzmefqhizpzed/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-kiwisolver-1.4.4-zhztqatkdiygep4euysdqohtcwhwpzxc/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-packaging-23.0-blxwysjulvhwyneaybdoivwv677ozlu4/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-pillow-9.5.0-j7fgafh5umvlmanofofgqliueaxjiood/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-pyparsing-3.0.9-6yq7wbtu2v3ucu7bjrqdjyz6sr4oselc/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-python-dateutil-2.8.2-tgxqgf2mm7wxskhiiljjx2ngyjax7u2a/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-six-1.16.0-5hs4nyajvpsyurprkjlu5cpcsb6chnf4/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-more-itertools-8.14.0-pgua2lnhmwi3g27osn45qrrwwyjh6lk4/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-pandas-1.5.3-exiq6pqewa7xryrzr777htetlrr4zprr/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-bottleneck-1.3.5-qlquzyi4zwlpalcixqq65zge53xkydft/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-numexpr-2.8.3-q6i5bcsimh26gdatyefdx22b7zen7yqs/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-pytz-2022.2.1-ebgng4ja4nqe2mhrkonpholimhxvqzoj/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-psutil-5.9.4-q5uf73s5haw7or4reykkg5xnmoe4cn4p/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-pyyaml-6.0-7rk53mwk4mcwuca36h4yckcazo74hqxi/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-seaborn-0.12.2-cjwoqaddyb7wrwafy26x5oazivm5ar76/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-sqlalchemy-1.4.45-kfluxm66wmggxtobzjg7ivblixzwcqbf/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-greenlet-2.0.2-gpd5as4h36htufxgtzgxqfuzt3at6wm2/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-versioneer-0.28-inx22ag3oliinugmpr6iox4viphdudp3/lib/python3.11/site-packages}
    prepend-path --delim {:} PYTHONPATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/py-zipp-3.8.1-xvvid6lqyefum7aaeajy3c33xtizhu6l/lib/python3.11/site-packages}
    setenv SURVEY_MPI_IMPLEMENTATION {openmpi}
    prepend-path --delim {:} PATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/python-3.11.1-pe7buux6r4o2ighv7rltvouao5utwh2g/bin}
    prepend-path --delim {:} PATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/papi-6.0.0.1-rl7uh6ygwpx6qhaxylptpyzkkgv5cxlp/bin}
    prepend-path --delim {:} PATH {/home/jgalarowicz/spack_releases/spack/opt/spack/linux-fedora37-icelake/gcc-12.3.1/libmonitor-2021.11.08-422cxh54dhbhsvfjv7bdekh5nrdiyslc/bin}
    append-path --delim {:} MANPATH {}

Thanks,
Jim G</div>


Discussion:
You should be able to use:

```yaml
autoload: direct
hide_implicits: true  # new in 0.21
```

The point being that `module load x` would load its dependencies recursively too, and each of them sets their own part of `PYTHONPATH`.

It looks like this is not documented well, so I'll do that next.
@haampie Thank you much for this answer!    I had to shift hide_implicits one level to the left otherwise spack was complaining:
```
==> Error: ConfigFormatError: /home/jgalarowicz/spack_02072024/spack/etc/spack/defaults/modules.yaml:48: Additional properties are not allowed ('hide_implicits' was unexpected)
```

When I shifted it to look like this, the build started and completed, but I see the module files generated and no PYTHONPATH commands generated in the survey module file.
What I want to see is no module files for the dependencies but PYTHONPATH commands for the python dependencies.  Is there a way to do that?   We are getting complaints at LANL that the module loads for the dependencies are interfering with the users modules.
```
   tcl:
      all:
        autoload: direct # then changed it to none
      hide_implicits: true  # new in 0.21

    # Default configurations if lmod is enabled
    lmod:
      all:
        autoload: direct # then changed it to none
      hide_implicits: true  # new in 0.21
      hierarchy:
        - mpi
```
> no PYTHONPATH commands generated in the survey module file.

It should have one for survey itself? Cause the package has `extends("python")`.

> We are getting complaints at LANL that the module loads for the dependencies are interfering with the users modules.

In what way exactly? Breaking / annoying / inconvenient? (There was a bug in Lmod where warnings "replacing module x with y" caused error exit codes, but should've been fixed in the meanwhile; warnings can be ignored)

I guess there are two approaches:

1. Autoloading is **enabled**, every module sets **only** its own environment variables, not those of its dependencies.
2. Autoloading is **disabled**, every module sets **all** environment variables of itself and its dependencies.

I was hoping that (1) would have no drawbacks in general, but maybe I'm mistaken?

The main upside of (1) is that you have clean and minimal module files, without duplication of environment variable values across modules, it's very composable, and you get reference counting from your module system.

Current approach (2) is not implemented to set environment variables from dependencies too, but I could be convinced to do so.
@haampie 

I've asked the person that is deploying survey on LANL systems to comment on the issues the users were having when the spack survey module loads all the dependent modules.   
Another person on my team sent me these statements about his experience on Darwin at LANL:
```
I also saw on Darwin that when it would try to load all the python modules – it would fail.

Besides – it also makes for a very messy module list for users where they see 20 additional modules loaded that they don’t understand.
```
We don't have issues with survey module loading when we use  autoload: none and spack generates the PYTHONPATH for the dependencies, but that functionality disappeared after v0.20.   
 
>  all the python modules – it would fail.

Fail, how?

Can you share some details about the module system type and version? Lmod with hierarchy or so?

> very messy module list

I though the hidden modules are actually hidden.
Here's the problem. When we provide Spack built packages via Spack created modulefiles with 'autoload: direct' (or anything other than autoload: none) - it works pretty well.  That is, until a code team with conflicting module files actually tries to use our software. In this case, (with autoload: direct) loading Survey would cause the reload of python and other packages the code team was building themselves. This would break either survey or the code itself, depending on the order in which modulefiles were loaded.
We can't account for what the user has loaded - so anything they load from our environment _MUST_ be stand-alone (well, outside of depending on a compiler and MPI).

Our deployments are using Spack 0.19 currently, and things seem to work with 'autoload:none'. If this breaks in newer versions of Spack, we're going to have to patch in the old behavior. 
Oh, and the `extends(python)` bit - we have to disable that for any package we build multiple versions of. It causes view conflicts (for obvious reasons).  I'd like to be able to just disable it in configs for certain packages, but Spack doesn't currently support that.

We also can't guarantee that the user of our survey package will have our Python loaded at all - so whatever dependencies Survey needs they must provided by the Survey modulefile itself.  This all works in Spack through RPATHs for almost everything else. There are, of course, possibilities for conflicts there too - setting PYTHONPATH might override a specific module the user is providing - but it's pretty likely to work anyway. 
Hm okay, that sounds brittle :)

So the issue comes down to:

1. There's a Spack and user `python` module.
2. The module system only allows one to be loaded concurrently 
3. Loading `survey` swaps in Spack's `python` module and warns about unloading theirs (or does it error?)

But then how would you want to see the "conflict" resolved?

Should `survey` embed `prepend_path PATH <path to spack's python>/bin`? I guess it has to?  But if that's all squashed into the `survey` module, it's gonna invalidate the user's `python` module too, just without warning?
It's incredibly fragile, but it's the best we've got at the moment.

The way I deal with this in any sort of python based software I write is to force the python either in the shebang or to wrap everything in a bash script that figures out the correct python, sets up the PYTHONPATH, and then starts the application.
________________________________
From: Harmen Stoppels ***@***.***>
Sent: Monday, February 12, 2024 6:41 AM
To: spack/spack ***@***.***>
Cc: Ferrell, Paul Steven ***@***.***>; Comment ***@***.***>
Subject: [EXTERNAL] Re: [spack/spack] PYTHONPATHs not set for python components after releases/v0.20 (Issue #42535)


Hm okay, that sounds brittle :)

So the issue comes down to:

  1.  There's a Spack and user python module.
  2.  The module system only allows one to be loaded concurrently
  3.  Loading survey swaps in Spack's python module and warns about unloading theirs (or does it error?)

But then how would you want to see the "conflict" resolved?

Should survey embed prepend_path PATH <path to spack's python>/bin? I guess it has to? But if that's all squashed into the top-level package, it's gonna invalidate the user's python module too, just without warning?

—
Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/spack/spack/issues/42535*issuecomment-1938701253__;Iw!!Bt8fGhp8LhKGRg!BzojhI94_wnfJRwuFoklEr7GdKRJtLZQXJblvqagdQTFq-Lui6gaYefp5EBpL6z0qvgUMwFpsoa-s_KpT_Tgyhi0EQ$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AMK6D5CUWKE4A5LEBAGLTNTYTILW3AVCNFSM6AAAAABC4USOS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZYG4YDCMRVGM__;!!Bt8fGhp8LhKGRg!BzojhI94_wnfJRwuFoklEr7GdKRJtLZQXJblvqagdQTFq-Lui6gaYefp5EBpL6z0qvgUMwFpsoa-s_KpT_RaLMrBRQ$>.
You are receiving this because you commented.Message ID: ***@***.***>

Yeah, you can fix it with executable wrappers that set environment variables -- that's what nix does. That's robust, but not very flexible, since users can't override those variables then.

I'm not sure how to make progress here. If you want a correctly working application, you need variables from the package and its dependencies. Autoloading makes that work, but if you don't want autoloading, squashing all variables of all dependencies into the top-level package's module sounds like the only way to guarantee that things work.

However, squashing variables into a module file also has downside, cause what if you have say

```
module load python
module load py-pkg # depends on python, so py-pkg has a superset of all variables that the python package has
module unload py-pkg
```

or

```
module load pkg-A-that-uses-mpich
module load pkg-B-that-uses-mpich # contains same mpich variables as pkg-A-that-uses-mpich
module unload pkg-B-that-uses-mpich
```

is that gonna break the currently loaded `python` / `pkg-A-that-uses-mpich` package because common/squashed variables are unset after unload?

I checked, and basically `prepend-path` commands are OK in both lmod and tcl modules. If two modules prepend the same path, it shows up only once in PATH, and the path only gets removed after unloading both.

But for `setenv` this is not the case. If two modules `setenv FOO bar`, then loading both and unloading _just one of the two_ unsets the variable, breaking the other loaded module.

`environment-modules` has https://modules.readthedocs.io/en/latest/modulefile.html#mfcmd-pushenv but it's only supported from version 5.1.

Also, it's somewhat unclear how to deal with module hierarchies.

Still feels like autoloading modules for dependencies is the most reliable way to work, even if the result is mildly annoying warnings about clashing modules.
It's not just mildly annoying, it breaks one application or the other.

We for the most part don't have modulefiles for individual python packages, so that's not a problem in our case.
________________________________
From: Harmen Stoppels ***@***.***>
Sent: Wednesday, February 14, 2024 5:44 AM
To: spack/spack ***@***.***>
Cc: Ferrell, Paul Steven ***@***.***>; Comment ***@***.***>
Subject: [EXTERNAL] Re: [spack/spack] PYTHONPATHs not set for python components after releases/v0.20 (Issue #42535)


Yeah, you can fix it with executable wrappers that set environment variables -- that's what nix does. That's robust, but not very flexible, since users can't override those variables then.

I'm not sure how to make progress here. If you want a correctly working application, you need variables from the package and its dependencies. Autoloading makes that work, but if you don't want autoloading, squashing all variables of all dependencies into the top-level package's module sounds like the only way to guarantee that things work.

However, squashing variables into a module file also has downside, cause what if you have say

module load python
module load py-pkg # depends on python, so py-pkg has a superset of all variables that the python package has
module unload py-pkg


or

module load mpich
module load pkg-that-uses-mpich # same, contains whatever is in mpich
module unload pkg-that-uses-mpich


is that gonna break the currently loaded python / mpich package because common/squashed variables are unset after unload?

I checked, and basically prepend-path commands are OK in both lmod and tcl modules. If two modules prepend the same path, it shows up only once in PATH, and the path only gets removed after unloading both.

But for setenv this is not the case. If two modules setenv FOO bar, then loading both and unloading just one of the two unsets the variable, breaking the other loaded module.

environment-modules has https://modules.readthedocs.io/en/latest/modulefile.html#mfcmd-pushenv<https://urldefense.com/v3/__https://modules.readthedocs.io/en/latest/modulefile.html*mfcmd-pushenv__;Iw!!Bt8fGhp8LhKGRg!G5Co4A7NWPaVWPhsuD_qqphQ3-QITn-eJvODLZHAU9nu3LsWumAH4O9-hFEnkWkTOc9s411FxLbdtQBdkw4aC6-WRw$> but it's only supported from version 5.1.

Also, it's somewhat unclear how to deal with module hierarchies.

Still feels like autoloading modules for dependencies is the most reliable way to work, even if the result is mildly annoying warnings about clashing modules.

—
Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/spack/spack/issues/42535*issuecomment-1943698485__;Iw!!Bt8fGhp8LhKGRg!G5Co4A7NWPaVWPhsuD_qqphQ3-QITn-eJvODLZHAU9nu3LsWumAH4O9-hFEnkWkTOc9s411FxLbdtQBdkw5HgrKGcQ$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AMK6D5F2YIX3OS4OIP2EGTLYTSWRDAVCNFSM6AAAAABC4USOS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBTGY4TQNBYGU__;!!Bt8fGhp8LhKGRg!G5Co4A7NWPaVWPhsuD_qqphQ3-QITn-eJvODLZHAU9nu3LsWumAH4O9-hFEnkWkTOc9s411FxLbdtQBdkw48DtcN-w$>.
You are receiving this because you commented.Message ID: ***@***.***>

I still don't understand what you mean by "breaks".
I just tested survey build using spack 0.20.0 and 0.22.0
- it did not fail loading (as it has in the past - but it takes much longer to load the module and the module list for a user is terribly confusing. - see below (0.20.0 first - appropriate, then 0.22.0)

module load openmpi/4.1.1-gcc_10.3.0 survey/1.0.9-openmpi.4

module list
Currently Loaded Modules:
  1) gcc/10.3.0   2) openmpi/4.1.1-gcc_10.3.0   3) survey/1.0.9-openmpi.4

now 0.22.0:

module load openmpi/4.1.1-gcc_10.3.0 survey/1.0.9.0126-gcc-10.3.0-27eia6d

module list
Currently Loaded Modules:
  1) gcc-runtime/10.3.0-gcc-10.3.0-5ktsmwi             39) libjpeg-turbo/3.0.0-gcc-10.3.0-urqk6ba
  2) zlib-ng/2.1.5-gcc-10.3.0-roar6gr                  40) py-pillow/10.0.0-gcc-10.3.0-q55mgku
  3) xz/5.4.1-gcc-10.3.0-zzoirsz                       41) openblas/0.3.20-gcc-10.3.0-qbclc4n
  4) util-linux-uuid/2.38.1-gcc-10.3.0-2hvydwx         42) py-numpy/1.26.3-gcc-10.3.0-v6wyew6
  5) tar/1.30-gcc-10.3.0-pr7riwu                       43) py-numexpr/2.8.4-gcc-10.3.0-ud7o7ge
  6) ncurses/6.4-gcc-10.3.0-a6keyjm                    44) py-bottleneck/1.3.7-gcc-10.3.0-qhudfbm
  7) readline/8.2-gcc-10.3.0-njefigm                   45) py-pandas/1.5.3-gcc-10.3.0-4boz7wv
  8) sqlite/3.43.2-gcc-10.3.0-4rbk3dl                  46) py-kiwisolver/1.4.5-gcc-10.3.0-thmptwm
  9) qhull/2020.2-gcc-10.3.0-mduj4o4                   47) py-fonttools/4.39.4-gcc-10.3.0-hkthfe4
 10) openssl/3.1.3-gcc-10.3.0-s5m6l4g                  48) py-cycler/0.11.0-gcc-10.3.0-3xl4lq5
 11) libiconv/1.17-gcc-10.3.0-kiji3pc                  49) py-contourpy/1.0.7-gcc-10.3.0-uwscldp
 12) libxml2/2.10.3-gcc-10.3.0-6yp2rtc                 50) libpng/1.6.39-gcc-10.3.0-kyp4tkt
 13) libxcrypt/4.4.35-gcc-10.3.0-4tufb67               51) freetype/2.11.1-gcc-10.3.0-mzynp4k
 14) libmd/1.0.4-gcc-10.3.0-wmmmziz                    52) py-matplotlib/3.8.2-gcc-10.3.0-jbngjv3
 15) libffi/3.4.4-gcc-10.3.0-glog4iz                   53) py-seaborn/0.12.2-gcc-10.3.0-yrb2oak
 16) libbsd/0.11.7-gcc-10.3.0-hyk2uny                  54) libyaml/0.2.5-gcc-10.3.0-mt4cobi
 17) bzip2/1.0.8-gcc-10.3.0-gpfp66z                    55) py-pyyaml/6.0-gcc-10.3.0-ccul737
 18) gettext/0.22.4-gcc-10.3.0-rsv5xnd                 56) py-psutil/5.9.5-gcc-10.3.0-vt62s7t
 19) gdbm/1.23-gcc-10.3.0-tzoknr4                      57) py-pluggy/1.0.0-gcc-10.3.0-vcusip2
 20) expat/2.5.0-gcc-10.3.0-x2lcy6a                    58) py-pathspec/0.11.1-gcc-10.3.0-y2hs3tn
 21) python/3.11.6-gcc-10.3.0-ekywzif                  59) py-more-itertools/9.1.0-gcc-10.3.0-jus7jvf
 22) py-zipp/3.17.0-gcc-10.3.0-7qotknu                 60) py-markupsafe/2.1.3-gcc-10.3.0-h43qr47
 23) py-versioneer/0.29-gcc-10.3.0-ksvl2zw             61) py-jinja2/3.1.2-gcc-10.3.0-647lkgj
 24) py-typing-extensions/4.8.0-gcc-10.3.0-xga4mbh     62) py-importlib-resources/5.12.0-gcc-10.3.0-pojbh7a
 25) py-trove-classifiers/2023.8.7-gcc-10.3.0-cl7c4mn  63) py-editables/0.3-gcc-10.3.0-5iysdp6
 26) py-tomli/2.0.1-gcc-10.3.0-mpzftoi                 64) py-hatchling/1.21.0-gcc-10.3.0-aszwapt
 27) py-greenlet/2.0.2-gcc-10.3.0-zncmwto              65) py-hatch-vcs/0.3.0-gcc-10.3.0-l77zapw
 28) py-sqlalchemy/1.4.49-gcc-10.3.0-or3zaha           66) py-humanize/4.6.0-gcc-10.3.0-mp2ncnn
 29) py-smmap/5.0.0-gcc-10.3.0-bf3lrwl                 67) py-gitdb/4.0.9-gcc-10.3.0-vq5hpnm
 30) py-six/1.16.0-gcc-10.3.0-kphbfk2                  68) py-gitpython/3.1.40-gcc-10.3.0-dovymqh
 31) py-setuptools/68.0.0-gcc-10.3.0-7gnyg7j           69) py-filelock/3.12.4-gcc-10.3.0-6bn75p4
 32) py-packaging/23.1-gcc-10.3.0-4dbcs6g              70) papi/7.1.0-gcc-10.3.0-cpjtrpw
 33) git/2.27.0-gcc-10.3.0-jkufmix                     71) llvm-openmp/12.0.1-gcc-10.3.0-roybdeu
 34) py-setuptools-scm/7.1.0-gcc-10.3.0-yixitgd        72) libmonitor/2021.11.08-gcc-10.3.0-nmunjbo
 35) py-pytz/2023.3-gcc-10.3.0-dvq6ht2                 73) gotcha/1.0.4-gcc-10.3.0-6jindcd
 36) py-python-dateutil/2.8.2-gcc-10.3.0-todrshi       74) survey/1.0.9.0126-gcc-10.3.0-27eia6d
 37) py-pyparsing/3.0.9-gcc-10.3.0-msgi5zn             75) gcc/10.3.0
 38) py-pybind11/2.11.1-gcc-10.3.0-uxleyba             76) openmpi/4.1.1-gcc_10.3.0





OK, I thought we were hiding hidden modules also in `module list`. Maybe that's only with `tcl`.

Resolving patch:
diff --git a/lib/spack/spack/modules/common.py b/lib/spack/spack/modules/common.py
--- a/lib/spack/spack/modules/common.py
+++ b/lib/spack/spack/modules/common.py
@@ -121,43 +121,26 @@ def update_dictionary_extending_lists(target, update):
             target[key] = update[key]
 
 
-def dependencies(spec, request="all"):
-    """Returns the list of dependent specs for a given spec, according to the
-    request passed as parameter.
+def dependencies(spec: spack.spec.Spec, request: str = "all") -> List[spack.spec.Spec]:
+    """Returns the list of dependent specs for a given spec.
 
     Args:
         spec: spec to be analyzed
-        request: either 'none', 'direct' or 'all'
+        request: one of "none", "run", "direct", "all"
 
     Returns:
-        list of dependencies
-
-        The return list will be empty if request is 'none', will contain
-        the direct dependencies if request is 'direct', or the entire DAG
-        if request is 'all'.
+        list of requested dependencies
     """
-    if request not in ("none", "direct", "all"):
-        message = "Wrong value for argument 'request' : "
-        message += "should be one of ('none', 'direct', 'all')"
-        raise tty.error(message + " [current value is '%s']" % request)
-
     if request == "none":
         return []
-
-    if request == "direct":
-        return spec.dependencies(deptype=("link", "run"))
-
-    # FIXME : during module file creation nodes seem to be visited multiple
-    # FIXME : times even if cover='nodes' is given. This work around permits
-    # FIXME : to get a unique list of spec anyhow. Do we miss a merge
-    # FIXME : step among nodes that refer to the same package?
-    seen = set()
-    seen_add = seen.add
-    deps = sorted(
-        spec.traverse(order="post", cover="nodes", deptype=("link", "run"), root=False),
-        reverse=True,
-    )
-    return [d for d in deps if not (d in seen or seen_add(d))]
+    elif request == "run":
+        return spec.dependencies(deptype=dt.RUN)
+    elif request == "direct":
+        return spec.dependencies(deptype=dt.RUN | dt.LINK)
+    elif request == "all":
+        return list(spec.traverse(order="topo", deptype=dt.LINK | dt.RUN, root=False))
+
+    raise ValueError(f'request "{request}" is not one of "none", "direct", "run", "all"')
 
 
 def merge_config_rules(configuration, spec):
diff --git a/lib/spack/spack/schema/modules.py b/lib/spack/spack/schema/modules.py
--- a/lib/spack/spack/schema/modules.py
+++ b/lib/spack/spack/schema/modules.py
@@ -34,7 +34,7 @@
 
 dictionary_of_strings = {"type": "object", "patternProperties": {r"\w[\w-]*": {"type": "string"}}}
 
-dependency_selection = {"type": "string", "enum": ["none", "direct", "all"]}
+dependency_selection = {"type": "string", "enum": ["none", "run", "direct", "all"]}
 
 module_file_configuration = {
     "type": "object",


Test patch: