Repository: networkx/networkx

Issue:
Error when importing networkx: "module 'importlib' has no attribute 'machinery'"
When importing networkx, the error `AttributeError: module 'importlib' has no attribute 'machinery'` occurs.

It seems like networkx is not importing `importlib.machinery`.

### Steps to Reproduce
```
$ sudo docker run -it --rm fedora:35
# dnf install -y python3 python3-pip
# pip3 install networkx
# python3 --version
Python 3.10.0rc2
# python3 -c "import importlib; print(dir(importlib))"
['_RELOADING', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__import__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '_bootstrap', '_bootstrap_external', '_imp', '_pack_uint32', '_unpack_uint32', 'find_loader', 'import_module', 'invalidate_caches', 'reload', 'sys', 'warnings']
# python3 -c "import networkx"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.10/site-packages/networkx/__init__.py", line 59, in <module>
    from networkx import utils
  File "/usr/local/lib/python3.10/site-packages/networkx/utils/__init__.py", line 1, in <module>
    from networkx.utils.misc import *
  File "/usr/local/lib/python3.10/site-packages/networkx/utils/misc.py", line 23, in <module>
    np = nx.lazy_import("numpy")
  File "/usr/local/lib/python3.10/site-packages/networkx/lazy_imports.py", line 138, in lazy_import
    tmp_loader = importlib.machinery.SourceFileLoader(module, path=None)
AttributeError: module 'importlib' has no attribute 'machinery'
# python3 -c "import importlib; import importlib.machinery; import networkx"
```

### Environment
Python version: Python 3.10.0rc2
NetworkX version: networkx-2.7


Discussion:
Very strange - maybe something is wrong with the `python3` package in the fedora image? It seems weird that you're getting 3.10rc2 (without specifying that explicitly).

FWIW I can't reproduce locally and CI never caught anything w/ Debian-based linux, macos, or windows.
I think this is due to which modules are loaded by default, which I guess is a platform-specific setting.

https://bugzilla.redhat.com/show_bug.cgi?id=2007664

Fedora 35:
```
$ python3 -c "import sys; print(sorted(sys.modules))"
['__main__', '_abc', '_codecs', '_collections_abc', '_frozen_importlib', '_frozen_importlib_external', '_imp', '_io', '_signal',
'_sitebuiltins', '_stat', '_thread', '_warnings', '_weakref', 'abc', 'builtins', 'codecs', 'encodings', 'encodings.aliases',
'encodings.utf_8', 'genericpath', 'io', 'marshal', 'os', 'os.path', 'posix', 'posixpath', 'site', 'stat', 'sys', 'time', 'zipimport']
```

Ubuntu 20.04:
```
$ python3 -c "import sys; print(sorted(sys.modules))"
['__main__', '_abc', '_bootlocale', '_codecs', '_collections', '_collections_abc', '_frozen_importlib', '_frozen_importlib_external',
'_functools', '_heapq', '_imp', '_io', '_locale', '_operator', '_signal', '_sitebuiltins', '_stat', '_thread', '_warnings', '_weakref', 'abc',
'apport_python_hook', 'builtins', 'codecs', 'collections', 'contextlib', 'encodings', 'encodings.aliases', 'encodings.latin_1',
'encodings.utf_8', 'functools', 'genericpath', 'heapq', 'importlib', 'importlib._bootstrap', 'importlib._bootstrap_external', 'importlib.abc',
'importlib.machinery', 'importlib.util', 'io', 'itertools', 'keyword', 'marshal', 'mpl_toolkits', 'operator', 'os', 'os.path', 'posix', 'posixpath',
'reprlib', 'site', 'sitecustomize', 'stat', 'sys', 'time', 'types', 'warnings', 'zipimport']
```
AFAIK `importlib.machinery` is part of the standard library and should be expected to be available in a Python installation. It seems like its absence might be an explicit decision in how Python is packaged in Fedora? I'm not sure there's a whole lot NetworkX can do about this, since the `SourceFileLoader` is an important component in the implementation of lazy loading in networkx
It is available, just not always imported by default. I think all that's needed is to add the line `import importlib.machinery` to `lazy_imports.py`. Usually it's safest to explicitly import all used modules rather than relying on them to be imported automatically or as a side-effect from another library.

For example, Python 3.9 on the current dev version of Ubuntu 22.04 doesn't include it by default either.

```
$ python3 -c "import importlib; importlib.machinery"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'machinery'
$ python3 -c "import importlib; import importlib.machinery; importlib.machinery"
$ echo $?
0
```
I had assumed that, since this never failed in CI nor in any developer's dev env that this was a problem with the Python packages in other distros... it appears that it's the opposite - the availability of `importlib.machinery` by default doesn't seem to be intentional:

https://github.com/python/cpython/blob/df9f7597559b6256924fcd3a1c3dc24cd5c5edaf/Lib/importlib/__init__.py#L2

I think you're right @stevenengler , we need to add an explicit import of the `machinery` module. Thanks for reporting and the follow-up research!
+1 person here to report on this issue causing problems with my CI.
Commenting so that I get auto-notified when the issue closes.

(btw, thanks for your development and continued maintenance of this great library!)
We plan to release a fix for this in 2.7.1 on Monday, March 7th.  Thanks for the bug report.
FWIW I set up docker and I can't reproduce this in any of the official images mentioned above, i.e. `fedora:35`,  `ubuntu:focal`, nor `ubuntu:jammy`. I'm not 100% convinced that this isn't some sort of python packaging problem in the images that are being used - the only evidence I've found to the contrary is the `__all__` I linked above from the `importlib` module, but that doesn't necessarily tell the whole story.

The "fix" is straightforward of course, but I'm just curious if anyone has any insight as to whether (and if so) why it's necessary?
I can provide the following data points:

[GitHub CI failling](https://github.com/tdene/synth_opt_adders/runs/5384343915?check_suite_focus=true) with nearly the same trace as OP, using [this workflow](https://github.com/tdene/synth_opt_adders/blob/main/.github/workflows/config.yml). The workflow `runs-on: ubuntu-latest` and uses CPython 3.10.2

On my personal machine with Python 3.8.10 and Ubuntu 20.04.4, I do not experience this issue, but I can replicate a very similar case:

```~$ python3
Python 3.8.10 (default, Nov 26 2021, 20:14:08) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib
>>> importlib.machinery
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'machinery'
>>> importlib.machinery
<module 'importlib.machinery' from '/usr/lib/python3.8/importlib/machinery.py'>
```

The same behavior persists with other built-in packages, such as urllib.request. Posts from [2015](https://web.archive.org/web/20160819073115/http://www.gossamer-threads.com/lists/python/python/1215580) and [2011](https://stackoverflow.com/questions/4871369/python-error-attributeerror-module-object-has-no-attribute) can be found discussing this.

So it seems related to choices made in `__init__.py`. I just learned this for the first time while writing this reply, but it looks like this is how Python behaves.
Now, why this causes errors in GitHub's CI, but not in local CI or manual use, I cannot guess.
> [GitHub CI failling](https://github.com/tdene/synth_opt_adders/runs/5384343915?check_suite_focus=true) with nearly the same trace as OP, using [this workflow](https://github.com/tdene/synth_opt_adders/blob/main/.github/workflows/config.yml).

Note that in this workflow the install step is done with `python` and the test-running step is done with `python3`, which may be the source of this particular issue. What happens if you modify these lines to:

```bash
python3 -m pip install --upgrade pip
python3 -m pip install .
```

NetworkX uses `ubuntu-latest` in its own CI workflows and this `importlib.machinery` issue has never cropped up.
Some other interesting behaviour: Which modules are loaded depends on whether you're in an interactive or non-interactive interpreter. For example on fedora:35, `machinery` is automatically added to the `importlib` namespace in an interactive interpreter, but not in a non-interactive one.

```
$ python3 -c "import importlib; importlib.machinery"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'machinery'

$ python3
Python 3.10.0 (default, Oct  4 2021, 00:00:00) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib
>>> importlib.machinery
<module 'importlib.machinery' from '/usr/lib64/python3.10/importlib/machinery.py'>
```

In a non-interactive interpreter, importing `importlib.util` loads `machinery` into the `importlib` namespace on Ubuntu 20.04, but not Fedora 35.

Ubuntu 20.04:

```
$ python3 -c "import importlib; importlib.machinery"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'machinery'
$ python3 -c "import importlib; import importlib.util; importlib.machinery"
$ echo $?
0
```

Fedora 35:

```
$ python3 -c "import importlib; importlib.machinery"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'machinery'
$ python3 -c "import importlib; import importlib.util; importlib.machinery"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'machinery'
```

Resolving patch:
diff --git a/networkx/lazy_imports.py b/networkx/lazy_imports.py
--- a/networkx/lazy_imports.py
+++ b/networkx/lazy_imports.py
@@ -1,5 +1,6 @@
 import importlib
 import importlib.util
+import inspect
 import types
 import os
 import sys
@@ -81,6 +82,24 @@ def __dir__():
     return __getattr__, __dir__, list(__all__)
 
 
+class DelayedImportErrorModule(types.ModuleType):
+    def __init__(self, frame_data, *args, **kwargs):
+        self.__frame_data = frame_data
+        super().__init__(*args, **kwargs)
+
+    def __getattr__(self, x):
+        if x in ("__class__", "__file__", "__frame_data"):
+            super().__getattr__(x)
+        else:
+            fd = self.__frame_data
+            raise ModuleNotFoundError(
+                f"No module named '{fd['spec']}'\n\n"
+                "This error is lazily reported, having originally occured in\n"
+                f'  File {fd["filename"]}, line {fd["lineno"]}, in {fd["function"]}\n\n'
+                f'----> {"".join(fd["code_context"]).strip()}'
+            )
+
+
 def lazy_import(fullname):
     """Return a lazily imported proxy for a module or library.
 
@@ -132,14 +151,18 @@ def myfunc():
     spec = importlib.util.find_spec(fullname)
 
     if spec is None:
-        # module not found - construct a DelayedImportErrorModule
-        spec = importlib.util.spec_from_loader(fullname, loader=None)
-        module = importlib.util.module_from_spec(spec)
-        tmp_loader = importlib.machinery.SourceFileLoader(module, path=None)
-        loader = DelayedImportErrorLoader(tmp_loader)
-        loader.exec_module(module)
-        # dont add to sys.modules. The module wasn't found.
-        return module
+        try:
+            parent = inspect.stack()[1]
+            frame_data = {
+                "spec": fullname,
+                "filename": parent.filename,
+                "lineno": parent.lineno,
+                "function": parent.function,
+                "code_context": parent.code_context,
+            }
+            return DelayedImportErrorModule(frame_data, "DelayedImportErrorModule")
+        finally:
+            del parent
 
     module = importlib.util.module_from_spec(spec)
     sys.modules[fullname] = module
@@ -148,24 +171,3 @@ def myfunc():
     loader.exec_module(module)
 
     return module
-
-
-class DelayedImportErrorLoader(importlib.util.LazyLoader):
-    def exec_module(self, module):
-        super().exec_module(module)
-        module.__class__ = DelayedImportErrorModule
-
-
-class DelayedImportErrorModule(types.ModuleType):
-    def __getattribute__(self, attr):
-        """Trigger a ModuleNotFoundError upon attribute access"""
-        spec = super().__getattribute__("__spec__")
-        # allows isinstance and type functions to work without raising error
-        if attr in ["__class__"]:
-            return super().__getattribute__("__class__")
-
-        raise ModuleNotFoundError(
-            f"Delayed Report: module named '{spec.name}' not found.\n"
-            "Reporting was Lazy -- delayed until module attributes accessed.\n"
-            f"Most likely, {spec.name} is not installed"
-        )


Test patch: