Repository: spack/spack

Issue:
Spack extensions have issues importing supplementary modules
### Steps to reproduce

Spack doc's state that extensions should be able to import modules that live in the 
`spack-[name]/[name]` directory, but that does not seem to be the case.
I've created a reproducer that illustrates this issue:
``` console
git clone --branch bug/import-issue https://github.com/psakievich/spack-scripting.git
spack config add config:extensions[$PWD/spack-scripting]
spack filter -h
```


### Error message

```console
$ spack filter -h
==> Error: No module named 'dummy'
[psakiev@hpws332 spack-scripting]$ spack --backtrace filter -h
Traceback (most recent call last):
  File "/ceeGPFS/fgs/psakiev/spack-manager-debug/spack/lib/spack/spack/cmd/__init__.py", line 120, in get_module
    module = __import__(module_name, fromlist=[pname, SETUP_PARSER, DESCRIPTION], level=0)
ModuleNotFoundError: No module named 'spack.cmd.filter'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/fgs/psakiev/spack-manager-debug/spack/bin/spack", line 52, in <module>
    sys.exit(main())
  File "/ceeGPFS/fgs/psakiev/spack-manager-debug/spack/lib/spack/spack_installable/main.py", line 37, in main
    sys.exit(spack.main.main(argv))
  File "/ceeGPFS/fgs/psakiev/spack-manager-debug/spack/lib/spack/spack/main.py", line 1066, in main
    return _main(argv)
  File "/ceeGPFS/fgs/psakiev/spack-manager-debug/spack/lib/spack/spack/main.py", line 1019, in _main
    return finish_parse_and_run(parser, cmd_name, args, env_format_error)
  File "/ceeGPFS/fgs/psakiev/spack-manager-debug/spack/lib/spack/spack/main.py", line 1025, in finish_parse_and_run
    command = parser.add_command(cmd_name)
  File "/ceeGPFS/fgs/psakiev/spack-manager-debug/spack/lib/spack/spack/main.py", line 358, in add_command
    module = spack.cmd.get_module(cmd_name)
  File "/ceeGPFS/fgs/psakiev/spack-manager-debug/spack/lib/spack/spack/cmd/__init__.py", line 123, in get_module
    module = spack.extensions.get_module(cmd_name)
  File "/ceeGPFS/fgs/psakiev/spack-manager-debug/spack/lib/spack/spack/extensions.py", line 158, in get_module
    module = load_command_extension(cmd_name, folder)
  File "/ceeGPFS/fgs/psakiev/spack-manager-debug/spack/lib/spack/spack/extensions.py", line 103, in load_command_extension
    module = importlib.import_module(module_name)
  File "/projects/cde/v3/cee/spack/opt/spack/linux-rhel7-x86_64/gcc-10.3.0/anaconda3-2021.05-hzr5w5v54sh3moi43gtl2fzdffebm2tf/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/fgs/psakiev/spack-manager-debug/spack-scripting/scripting/cmd/filter.py", line 28, in <module>
    import dummy
ModuleNotFoundError: No module named 'dummy'
```

### Information on your system

* **Spack:** 0.22.0.dev0 (d06a102e69c3b546b50e5dd4290aae9654516ae6)
* **Python:** 3.8.8
* **Platform:** linux-rhel7-cascadelake
* **Concretizer:** clingo

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
Let me improve documentation on this. The modules can be imported, but they are currently namespaced under 
```
spack.extensions.scripting  # spack.extensions.<name-of-the-extension>
```
to avoid name clashes with other extensions. This means that if you apply:
```diff
diff --git a/scripting/cmd/filter.py b/scripting/cmd/filter.py
index a75dfa3..ba1f177 100644
--- a/scripting/cmd/filter.py
+++ b/scripting/cmd/filter.py
@@ -25,7 +25,7 @@
 
 import argparse
 import collections
-import dummy
+from spack.extensions.scripting import dummy
 import sys
 
 import spack
diff --git a/scripting/dummy.py b/scripting/dummy.py
index 61658c5..78c9b36 100644
--- a/scripting/dummy.py
+++ b/scripting/dummy.py
@@ -1,2 +1,2 @@
 def hello_world():
-    print "hello world!"
+    print("hello world!")

```
The module will load.
Even better, you can do:
```python
from ..dummy import hello_world
```
in the command itself.

Resolving patch:
diff --git a/lib/spack/spack/extensions.py b/lib/spack/spack/extensions.py
--- a/lib/spack/spack/extensions.py
+++ b/lib/spack/spack/extensions.py
@@ -6,11 +6,13 @@
 for Spack's command extensions.
 """
 import difflib
+import glob
 import importlib
 import os
 import re
 import sys
 import types
+from typing import List
 
 import llnl.util.lang
 
@@ -75,6 +77,15 @@ def load_command_extension(command, path):
     if not os.path.exists(cmd_path):
         return None
 
+    ensure_extension_loaded(extension, path=path)
+
+    module = importlib.import_module(module_name)
+    sys.modules[module_name] = module
+
+    return module
+
+
+def ensure_extension_loaded(extension, *, path):
     def ensure_package_creation(name):
         package_name = "{0}.{1}".format(__name__, name)
         if package_name in sys.modules:
@@ -100,10 +111,22 @@ def ensure_package_creation(name):
     ensure_package_creation(extension)
     ensure_package_creation(extension + ".cmd")
 
-    module = importlib.import_module(module_name)
-    sys.modules[module_name] = module
 
-    return module
+def load_extension(name: str) -> str:
+    """Loads a single extension into the 'spack.extensions' package.
+
+    Args:
+        name: name of the extension
+    """
+    extension_root = path_for_extension(name, paths=get_extension_paths())
+    ensure_extension_loaded(name, path=extension_root)
+    commands = glob.glob(
+        os.path.join(extension_root, extension_name(extension_root), "cmd", "*.py")
+    )
+    commands = [os.path.basename(x).rstrip(".py") for x in commands]
+    for command in commands:
+        load_command_extension(command, extension_root)
+    return extension_root
 
 
 def get_extension_paths():
@@ -125,7 +148,7 @@ def get_command_paths():
     return command_paths
 
 
-def path_for_extension(target_name, *paths):
+def path_for_extension(target_name: str, *, paths: List[str]) -> str:
     """Return the test root dir for a given extension.
 
     Args:


Test patch:
diff --git a/lib/spack/spack/cmd/unit_test.py b/lib/spack/spack/cmd/unit_test.py
--- a/lib/spack/spack/cmd/unit_test.py
+++ b/lib/spack/spack/cmd/unit_test.py
@@ -227,9 +227,7 @@ def unit_test(parser, args, unknown_args):
     # has been used, then test that extension.
     pytest_root = spack.paths.spack_root
     if args.extension:
-        target = args.extension
-        extensions = spack.extensions.get_extension_paths()
-        pytest_root = spack.extensions.path_for_extension(target, *extensions)
+        pytest_root = spack.extensions.load_extension(args.extension)
 
     # pytest.ini lives in the root of the spack repository.
     with llnl.util.filesystem.working_dir(pytest_root):