Repository: astropy/astropy

Issue:
API: Allow dir(astropy) to list all subpackage names
### What is the problem this feature will solve?

When one is offline and unable to check online docs, one wishes to use `dir(astropy)` to list all available subpackages for discovery.

### Describe the desired outcome

`dir(astropy)` to list things like `"stats"`.

As of `astropy` 7.1.dev, it only listed the following:

```python
['__bibtex__',
 '__builtins__',
 '__cached__',
 '__citation__',
 '__dir_inc__',
 '__doc__',
 '__file__',
 '__loader__',
 '__name__',
 '__package__',
 '__path__',
 '__spec__',
 '__version__',
 'astronomical_constants',
 'conf',
 'config',
 'extern',
 'find_api_page',
 'log',
 'logger',
 'online_docs_root',
 'online_help',
 'physical_constants',
 'system_info',
 'test',
 'tests',
 'utils',
 'version']
```

### Additional context

* Originally requested in https://github.com/astropy/astropy/pull/17587#issuecomment-2566540027
* Not sure how much this would be addressed by https://github.com/astropy/astropy-APEs/pull/85

Discussion:
github-actions[bot]: Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

  - [ ] Do the proposed changes actually accomplish desired goals?
  - [ ] Do the proposed changes follow the [Astropy coding guidelines](https://docs.astropy.org/en/latest/development/codeguide.html)?
  - [ ] Are tests added/updated as required? If so, do they follow the [Astropy testing guidelines](https://docs.astropy.org/en/latest/development/testguide.html)?
  - [ ] Are docs added/updated as required? If so, do they follow the [Astropy documentation guidelines](https://docs.astropy.org/en/latest/development/docguide.html)?
  - [ ] Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for [rebase](https://docs.astropy.org/en/latest/development/development_details.html#rebase-if-necessary) and [squash](https://docs.astropy.org/en/latest/development/development_details.html#squash-if-necessary).
  - [ ] Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the [bot](https://docs.astropy.org/en/latest/development/development_details.html#pre-commit).
  - [ ] Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
  - [ ] Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
  - [ ] At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) *before* merge.
pllim: @celik-muhammed are you satisfied with this proposed solution?
pllim: pyinstaller is not happy, hmm, why?

```
______________________________ test_find_mod_objs ______________________________

    def test_find_mod_objs():
        deprecation_message = (
            "^The find_mod_objs function is deprecated and may be removed in "
            r"a future version\.$"
        )
        with pytest.warns(AstropyDeprecationWarning, match=deprecation_message):
>           lnms, fqns, objs = find_mod_objs("astropy")

astropy_tests/utils/tests/test_introspection.py:58: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
astropy/utils/decorators.py:141: in deprecated_func
    ???
astropy/utils/introspection.py:341: in find_mod_objs
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

.0 = <list_iterator object at 0x7f02df7dfb50>

>   ???
E   KeyError: 'convolution'

astropy/utils/introspection.py:341: KeyError
```
pllim: Maybe this needs patching too?

https://github.com/astropy/astropy/blob/80aa67d2cc7ddb41a63d00d231b2792aafca77fa/astropy/utils/introspection.py#L341
mhvk: Nice catch on `find_mod_objs` - I pushed a fix... (clearly, there is a reason to have deprecated it -- it was written before module-level `__getattr__`...)
mhvk: > Hmm seems to break RTD somehow.
> 
> ```
> <unknown>:15: WARNING: py:obj reference target not found: get_name [ref.obj]
> <unknown>:15: WARNING: py:obj reference target not found: name [ref.obj]
> ```

Seems to be because of a bad changelog entry, from #17503. Not sure why that didn't get caught in testing before, though...
mhvk: I added two commits that should fix the RTD problems. Since these are not directly related to this PR (though somehow it exposes the problems!?), probably best not to squash-merge this PR.
mhvk: p.s. Because this fixes a previous changelog entry, the bot fails - so I put `skip-changelog-checks` - this part of course passed before, so that should be fine.
pllim: Hmm

```
  >       assert cp.returncode == 0
  E       AttributeError: 'int' object has no attribute 'returncode'
```

Patch:
diff --git a/astropy/__init__.py b/astropy/__init__.py
index a7f600ba4c..73edd5f7b0 100644
--- a/astropy/__init__.py
+++ b/astropy/__init__.py
@@ -11,6 +11,55 @@ from pathlib import Path
 
 from astropy.utils.system_info import system_info
 
+__all__ = [  # noqa: RUF100, RUF022
+    "__version__",
+    "__bibtex__",
+    # Subpackages (mostly lazy-loaded)
+    "config",
+    "constants",
+    "convolution",
+    "coordinates",
+    "cosmology",
+    "io",
+    "modeling",
+    "nddata",
+    "samp",
+    "stats",
+    "table",
+    "tests",
+    "time",
+    "timeseries",
+    "uncertainty",
+    "units",
+    "utils",
+    "visualization",
+    "wcs",
+    # Functions
+    "test",
+    "log",
+    "find_api_page",
+    "online_help",
+    "online_docs_root",
+    "conf",
+    "physical_constants",
+    "astronomical_constants",
+    "system_info",
+]
+
+
+def __getattr__(attr):
+    if attr in __all__:
+        from importlib import import_module
+
+        return import_module("astropy." + attr)
+
+    raise AttributeError(f"module 'astropy' has no attribute {attr!r}")
+
+
+def __dir__():
+    return sorted(set(globals()).union(__all__))
+
+
 from .version import version as __version__
 
 # The location of the online documentation for astropy
@@ -198,30 +247,14 @@ def online_help(query):
     webbrowser.open(url)
 
 
-__dir_inc__ = [
-    "__version__",
-    "__githash__",
-    "__bibtex__",
-    "test",
-    "log",
-    "find_api_page",
-    "online_help",
-    "online_docs_root",
-    "conf",
-    "physical_constants",
-    "astronomical_constants",
-    "system_info",
-]
-
-
 from types import ModuleType as __module_type__
 
-# Clean up top-level namespace--delete everything that isn't in __dir_inc__
+# Clean up top-level namespace--delete everything that isn't in __all__
 # or is a magic attribute, and that isn't a submodule of this package
 for varname in dir():
     if not (
         (varname.startswith("__") and varname.endswith("__"))
-        or varname in __dir_inc__
+        or varname in __all__
         or (
             varname[0] != "_"
             and isinstance(locals()[varname], __module_type__)
diff --git a/astropy/utils/introspection.py b/astropy/utils/introspection.py
index a8110e6702..626c067778 100644
--- a/astropy/utils/introspection.py
+++ b/astropy/utils/introspection.py
@@ -338,9 +338,9 @@ def find_mod_objs(modname, onlylocals=False):
     mod = import_module(modname)
 
     if hasattr(mod, "__all__"):
-        pkgitems = [(k, mod.__dict__[k]) for k in mod.__all__]
+        pkgitems = [(k, getattr(mod, k)) for k in mod.__all__]
     else:
-        pkgitems = [(k, mod.__dict__[k]) for k in dir(mod) if k[0] != "_"]
+        pkgitems = [(k, getattr(mod, k)) for k in dir(mod) if k[0] != "_"]
 
     # filter out modules and pull the names and objs out
     ismodule = inspect.ismodule
diff --git a/docs/changes/17598.other.rst b/docs/changes/17598.other.rst
new file mode 100644
index 0000000000..5cd776967e
--- /dev/null
+++ b/docs/changes/17598.other.rst
@@ -0,0 +1,4 @@
+After ``import astropy``, ``dir(astropy)`` will now list all subpackages,
+including those that have not yet been loaded. This also means tab
+completion will work as expected (e.g., ``from astropy.coo<TAB>`` will
+expand to ``from astropy.coordinates``).
diff --git a/docs/changes/coordinates/17503.api.rst b/docs/changes/coordinates/17503.api.rst
index 1f34e41034..27b1f9e25b 100644
--- a/docs/changes/coordinates/17503.api.rst
+++ b/docs/changes/coordinates/17503.api.rst
@@ -1,2 +1,2 @@
-On representations the method `get_name` has been deprecated in favor of the class-level
-attribute `name`. The method will be removed in a future release.
+On representations the method ``get_name`` has been deprecated in favor of the class-level
+attribute ``name``. The method will be removed in a future release.
diff --git a/docs/nitpick-exceptions b/docs/nitpick-exceptions
index e9563b5ed9..219e23bf57 100644
--- a/docs/nitpick-exceptions
+++ b/docs/nitpick-exceptions
@@ -74,6 +74,7 @@ py:class None.  Remove all items from D.
 py:class a set-like object providing a view on D's items
 py:class a set-like object providing a view on D's keys
 py:class None.  Update D from dict/iterable E and F.
+py:class None.  Update D from mapping/iterable E and F.
 py:class an object providing a view on D's values
 py:class a shallow copy of D
 


Test patch:
diff --git a/astropy/tests/tests/test_imports.py b/astropy/tests/tests/test_imports.py
index 009b0278e8..fb62bf4105 100644
--- a/astropy/tests/tests/test_imports.py
+++ b/astropy/tests/tests/test_imports.py
@@ -1,6 +1,15 @@
 # Licensed under a 3-clause BSD style license - see LICENSE.rst
 
 import pkgutil
+import subprocess
+import sys
+from pathlib import Path
+from textwrap import dedent
+from types import ModuleType
+
+import pytest
+
+import astropy
 
 
 def test_imports():
@@ -32,3 +41,38 @@ def test_toplevel_namespace():
     assert "log" in d
     assert "test" in d
     assert "sys" not in d
+
+
+def test_toplevel_lazy_imports():
+    # Check that subpackages are loaded on demand.
+    cmd = dedent("""
+    import astropy, sys
+    assert 'astropy.units' not in sys.modules
+    astropy.units
+    assert 'astropy.units' in sys.modules
+    """)
+    cp = subprocess.check_call([sys.executable, "-c", cmd])
+    assert cp == 0
+
+
+def test_toplevel_attribute_error():
+    # Ensure that our __getattr__ does not leak an import error or so.
+    with pytest.raises(AttributeError, match="module 'astropy' has no"):
+        astropy.nonsense
+
+
+def test_completeness_toplevel__all__():
+    # Implicitly check that all items in __all__ exist.
+    all_items = {getattr(astropy, attr) for attr in astropy.__all__}
+    # Verify that the list of modules in __all__ is complete.
+    module_names = {
+        item.__name__.partition(".")[2]
+        for item in all_items
+        if isinstance(item, ModuleType)
+    }
+    module_dirs = {
+        f.name
+        for f in Path(astropy.__file__).parent.glob("[a-z]*")
+        if f.is_dir() and f.name != "extern"
+    }
+    assert module_names == module_dirs