Repository: napari/napari

Issue:
napari.layers cannot be found on master
## 🐛 Bug

Probably introduced in #2662, one can no longer do:

```python
import numpy as np
import napari

image = napari.layers.Image(np.random.random((500, 500)))
```

Trying this on master, I get:

```pytb
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~/projects/napari/napari/__init__.py in __getattr__(name)
     75     try:
---> 76         return locals()[name]
     77     except KeyError:

KeyError: 'layers'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
<ipython-input-2-c7cb80ce1324> in <module>
----> 1 layer = napari.layers.Image(np.random.random((500, 500)), scale=[10, 10])

~/projects/napari/napari/__init__.py in __getattr__(name)
     76         return locals()[name]
     77     except KeyError:
---> 78         raise AttributeError(f"module 'napari' has no attribute {name!r}")

AttributeError: module 'napari' has no attribute 'layers'
```

`from napari import layers` works fine, so the module is there. :grimacing: 

This is an API breakage so it unfortunately needs to be fixed before 0.4.9.


Discussion:
>This is an API breakage so it unfortunately needs to be fixed before 0.4.9.

Indeed :) 

---

The bug is not strictly `high-severity` since there is a workaround. The severity is pretty low when there is a workaround however annoying the bug may be.

Higher severities are associated with napari actually breaking, segfaulting without any means for the users to have a workaround to get napari running 🤓 


On another note, I have seen this happen on some other packages (even python standard library itself I think) so the question I have (in general) once you import the top module, should you be able to access any submodule from it 🤔 ?
yeah, we've never exported `layers` in the napari namespace... so this has worked in the past due to other inner modules importing it.

> once you import the top module, should you be able to access any submodule from it

not automatically, it depends on what else imports it.  For instance, take this package structure:

```
my_module/
├─ __init__.py
├─ a.py
├─ b.py
```
where `a.py` and `b.py` are empty and 

```python
# __init__.py
from . import a
```

```python
In [1]: import my_module

In [2]: my_module.a
Out[2]: <module 'my_module.a' from '~/my_module/a.py'>

In [3]: my_module.b
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-5cb7b5e01231> in <module>
----> 1 my_module.b

AttributeError: module 'my_module' has no attribute 'b'
```

however, if within `a.py` you import `b`:

```python
# a.py
from . import b
```

then it works as @jni was expecting in the first post:
```python
In [1]: import my_module

In [2]: dir(my_module)
Out[2]:
['__builtins__',
 '__cached__',
 '__doc__',
 '__file__',
 '__loader__',
 '__name__',
 '__package__',
 '__path__',
 '__spec__',
 'a',
 'b']

In [3]: my_module.b
Out[3]: <module 'my_module.b' from '~/my_module/b.py'>
```

So basically, the fact that it worked before was an indirect import of `napari.layers` somewhere inside some other module that we do export in `napari.__init__`.  The `__getattr__` approach definitely requires you to be explicit about the namespace of the top level package ... but I'm not sure that's a bad thing?
>The __getattr__ approach definitely requires you to be explicit about the namespace of the top level package ... but I'm not sure that's a bad thing?

Well it is a convenience, and it was like that before (for other reasons) but I do not think being explicit about the imports is bad either.
ok.  I've played with this a little and while I can fix `layers` pretty easily with the `__getattr__`, it's not as easy for all of the things (`experimental` for instance is giving me circular import problems with the approaches I've tried).  So if we want to get back the behavior of being able to import `napari` and then access all of the top level modules implicitly, we may want to roll back #2662 as well... 

which will of course bring back the double import problem in `__main__`
@jni, are you still feeling pretty strongly about keeping the cli module inside `napari.__main__`? 
>So if we want to get back the behavior of being able to import napari and then access all of the top level modules implicitly, we may want to roll back #2662 as well...

[PR](https://github.com/napari/napari/pull/2760) on settings is now depending on that behavior :(


> I can fix layers pretty easily with the __getattr__, it's not as easy for all of the things (experimental for instance is giving me circular import problems with the approaches I've tried)

Hmm, that sucks, but I suspect there is a solution somewhere. Don't have time to look at this in detail right now but there is discussion happening about this pattern at the scipy ecosystem level — here is the [scikit-image PR](https://github.com/scikit-image/scikit-image/pull/5101) and the [SPEC discussion](https://discuss.scientific-python.org/t/spec-1-lazy-loading-for-submodules/25/3). As you can see the skimage PR went for aaaaaages so there's certainly a lot of subtlety, but it might be "done" now.
> yeah, we've never exported layers in the napari namespace... so this has worked in the past due to other inner modules importing it.

I did wonder about that, but I was excited to see it work in my usage and then got used to it! 😂 
@goanpeca feel free to correct my label usage. 😊 
i'd say lets just roll it back, and deal with the slow imports some other way
I'm sure it can "be done" ... but after playing with it for 30 minutes or so, it started to feel like a rabbit hole of magic that's going to be a continual frustration (and it also exposed that we've been relying on python imports and `sys.modules` being even cooler than we knew behind the scenes for us).
Just chatting with @stefanv now, he says all the magic is here:

https://github.com/scikit-image/scikit-image/pull/5101/files#diff-cd88cca5d4b70042d79332dc7132ad42a8ea07929b479cf64be5bddb1f7f0548R5

and here:

https://github.com/scikit-image/scikit-image/pull/5101/files#diff-5cf0c3839f5afeb18eb878337af2a47e09d91e50080d007d9c3f5123659ce2d8R96-R97

so we should be able to copy that `lazy.py` and use it. Imma give it a try.

Resolving patch:
diff --git a/napari/__init__.py b/napari/__init__.py
--- a/napari/__init__.py
+++ b/napari/__init__.py
@@ -1,31 +1,41 @@
+import os
+
 try:
     from ._version import version as __version__
 except ImportError:
     __version__ = "not-installed"
 
+# Allows us to use pydata/sparse arrays as layer data
+os.environ.setdefault('SPARSE_AUTO_DENSIFY', '1')
+del os
+
 # Add everything that needs to be accessible from the napari namespace here.
-__all__ = [
+_proto_all_ = [
     '__version__',
-    'gui_qt',
-    'notification_manager',
-    'run',
-    'save_layers',
-    'sys_info',
-    'view_image',
-    'view_labels',
-    'view_path',
-    'view_points',
-    'view_shapes',
-    'view_surface',
-    'view_tracks',
-    'view_vectors',
-    'Viewer',
+    'components',
+    'experimental',
+    'layers',
+    'qt',
+    'utils',
 ]
 
-
-def __dir__():
-    return __all__
-
+_submod_attrs = {
+    '_event_loop': ['gui_qt', 'run'],
+    'plugins.io': ['save_layers'],
+    'utils': ['sys_info'],
+    'utils.notifications': ['notification_manager'],
+    'view_layers': [
+        'view_image',
+        'view_labels',
+        'view_path',
+        'view_points',
+        'view_shapes',
+        'view_surface',
+        'view_tracks',
+        'view_vectors',
+    ],
+    'viewer': ['Viewer'],
+}
 
 # All imports in __init__ are hidden inside of `__getattr__` to prevent
 # importing the full chain of packages required when calling `import napari`.
@@ -36,43 +46,9 @@ def __dir__():
 # potential to take a second or more, so we definitely don't want to import it
 # just to access the CLI (which may not actually need any of the imports)
 
+from ._lazy import install_lazy
 
-def __getattr__(name):
-    # this unused import is here to fix a very strange bug.
-    # there is some mysterious magical goodness in scipy stats that needs
-    # to be imported early.
-    # see: https://github.com/napari/napari/issues/925
-    # see: https://github.com/napari/napari/issues/1347
-    from scipy import stats  # noqa: F401
-
-    # register napari object types with magicgui if it is installed
-    from .utils import _magicgui, sys_info
-    from .utils.notifications import notification_manager
-    from .viewer import Viewer
-
-    # This must come before .plugins
-    _magicgui.register_types_with_magicgui()
-
-    from ._event_loop import gui_qt, run
-    from .plugins.io import save_layers
-    from .view_layers import (  # type: ignore
-        view_image,
-        view_labels,
-        view_path,
-        view_points,
-        view_shapes,
-        view_surface,
-        view_tracks,
-        view_vectors,
-    )
-
-    del _magicgui
-    del stats
-    import os
-
-    os.environ.setdefault('SPARSE_AUTO_DENSIFY', '1')
-
-    try:
-        return locals()[name]
-    except KeyError:
-        raise AttributeError(f"module 'napari' has no attribute {name!r}")
+__getattr__, __dir__, __all__ = install_lazy(
+    __name__, _proto_all_, _submod_attrs
+)
+del install_lazy
diff --git a/napari/_lazy.py b/napari/_lazy.py
new file mode 100644
--- /dev/null
+++ b/napari/_lazy.py
@@ -0,0 +1,67 @@
+import importlib
+import importlib.util
+
+
+def install_lazy(module_name, submodules=None, submod_attrs=None):
+    """Install lazily loaded submodules, and functions or other attributes.
+
+    Parameters
+    ----------
+    module_name : str
+        Typically use __name__.
+    submodules : set
+        List of submodules to install.
+    submod_attrs : dict
+        Dictionary of submodule -> list of attributes / functions.
+        These attributes are imported as they are used.
+
+    Returns
+    -------
+    __getattr__, __dir__, __all__
+
+    """
+    if submod_attrs is None:
+        submod_attrs = {}
+
+    if submodules is None:
+        submodules = set()
+    else:
+        submodules = set(submodules)
+
+    attr_to_modules = {
+        attr: mod for mod, attrs in submod_attrs.items() for attr in attrs
+    }
+
+    __all__ = list(submodules | attr_to_modules.keys())
+
+    def __getattr__(name):
+        # this unused import is here to fix a very strange bug.
+        # there is some mysterious magical goodness in scipy stats that needs
+        # to be imported early.
+        # see: https://github.com/napari/napari/issues/925
+        # see: https://github.com/napari/napari/issues/1347
+        from scipy import stats  # noqa: F401
+
+        # This must come before .plugins
+        from .utils import _magicgui
+
+        _magicgui.register_types_with_magicgui()
+        del _magicgui
+
+        if name in submodules:
+            module = importlib.import_module(f'{module_name}.{name}')
+            del stats
+            return module
+        elif name in attr_to_modules:
+            submod = importlib.import_module(
+                f'{module_name}.{attr_to_modules[name]}'
+            )
+            del stats
+            return getattr(submod, name)
+        else:
+            raise AttributeError(f'No {module_name} attribute {name}')
+
+    def __dir__():
+        return __all__
+
+    return __getattr__, __dir__, __all__


Test patch: