Repository: yt-project/yt

Issue:
TST: CI instability (xref upstream breakage from setuptools+numpy+astropy)
### Bug report

**Bug summary**

`yt/visualization/tests/test_fits_image.py` started failing on certain platforms a few hours ago, see for instance
https://github.com/yt-project/yt/actions/runs/3504488379/jobs/5870201144

It seems that the on-demand-import wrapper around astropy is crashing the test with
```
ImportError: This functionality requires the astropy package to be installed.
```
However astropy is indeed installed in those jobs (see the logs). There were no recent changes (less than 24h) in the test itself or the `fits_image.py` module, astropy wasn't upgraded since the last successful job, and I haven't been able to reproduce this problem locally thus far.


Discussion:
So, another library that I maintain, unrelated to yt, began failing tests with a suspiciously similar error:
```
E       ImportError: Convolution C extension is missing. Try re-building astropy.
```

And pulling the thread leads to https://github.com/pypa/setuptools/issues/3693
This issue affects a _lot_ of projects downstream of numpy, so the best course of action for us is probably to just wait it out.

That said, if yt on-demand-imports mechanism didn't swallow the initial `ImportError` message completely, it would have saved me a couple hours of vain research, so I guess that's an easy actionable on our side that could close this issue (when CI is stable again)
a final note for now: in Python 3.11, we could leverage this feature: https://docs.python.org/3/whatsnew/3.11.html#pep-678-exceptions-can-be-enriched-with-notes

This would be useful *if* we can manage to keep things tidy on earlier versions too.

Resolving patch:
diff --git a/yt/utilities/on_demand_imports.py b/yt/utilities/on_demand_imports.py
--- a/yt/utilities/on_demand_imports.py
+++ b/yt/utilities/on_demand_imports.py
@@ -1,7 +1,7 @@
 import sys
 from functools import wraps
 from importlib.util import find_spec
-from typing import Type
+from typing import Optional, Type
 
 
 class NotAModule:
@@ -11,11 +11,27 @@ class NotAModule:
     package installed.
     """
 
-    def __init__(self, pkg_name):
+    def __init__(self, pkg_name, exc: Optional[BaseException] = None):
         self.pkg_name = pkg_name
-        self.error = ImportError(
-            f"This functionality requires the {self.pkg_name} package to be installed."
+        self._original_exception = exc
+        error_note = (
+            f"Something went wrong while trying to lazy-import {pkg_name}. "
+            f"Please make sure that {pkg_name} is properly installed.\n"
+            "If the problem persists, please file an issue at "
+            "https://github.com/yt-project/yt/issues/new"
         )
+        self.error: BaseException
+        if exc is None:
+            self.error = ImportError(error_note)
+        elif sys.version_info >= (3, 11):
+            exc.add_note(error_note)
+            self.error = exc
+        else:
+            # mimick Python 3.11 behaviour:
+            # preserve error message and traceback
+            self.error = type(exc)(f"{exc!s}\n{error_note}").with_traceback(
+                exc.__traceback__
+            )
 
     def __getattr__(self, item):
         raise self.error
@@ -24,7 +40,10 @@ def __call__(self, *args, **kwargs):
         raise self.error
 
     def __repr__(self) -> str:
-        return f"NotAModule({self.pkg_name!r})"
+        if self._original_exception is None:
+            return f"NotAModule({self.pkg_name!r})"
+        else:
+            return f"NotAModule({self.pkg_name!r}, {self._original_exception!r})"
 
 
 class OnDemand:
@@ -57,8 +76,8 @@ def safe_import(func):
     def inner(self):
         try:
             return func(self)
-        except ImportError:
-            return self._default_factory(self._name)
+        except ImportError as exc:
+            return self._default_factory(self._name, exc)
 
     return inner
 


Test patch:
diff --git a/yt/utilities/tests/test_on_demand_imports.py b/yt/utilities/tests/test_on_demand_imports.py
--- a/yt/utilities/tests/test_on_demand_imports.py
+++ b/yt/utilities/tests/test_on_demand_imports.py
@@ -27,10 +27,24 @@ def spam(self):
     _bacon = Bacon_imports()
     with pytest.raises(
         ImportError,
-        match=r"This functionality requires the Bacon package to be installed\.",
-    ):
+        match=r"No module named 'Bacon'",
+    ) as excinfo:
         _bacon.spam()
 
+    # yt should add information to the original error message
+    # but this done slightly differently in Python>=3.11
+    # (using exception notes), so we can't just match the error message
+    # directly. Instead this implements a Python-version agnostic check
+    # that the user-visible error message is what we expect.
+    complete_error_message = excinfo.exconly()
+    assert complete_error_message == (
+        "ModuleNotFoundError: No module named 'Bacon'\n"
+        "Something went wrong while trying to lazy-import Bacon. "
+        "Please make sure that Bacon is properly installed.\n"
+        "If the problem persists, please file an issue at "
+        "https://github.com/yt-project/yt/issues/new"
+    )
+
 
 def test_class_invalidation():
     with pytest.raises(