Repository: scipy/scipy
Pull request: DEP: optimize: improve deprecation of private modules named without underscore prefix
<!-- 
Thanks for contributing a pull request! Please ensure that
your PR satisfies the checklist before submitting:
http://scipy.github.io/devdocs/dev/contributor/development_workflow.html#checklist-before-submitting-a-pr

Also, please name and describe your PR as you would write a
commit message:
http://scipy.github.io/devdocs/dev/contributor/development_workflow.html#writing-the-commit-message

Depending on your changes, you can skip CI operations and save time and energy: 
http://scipy.github.io/devdocs/dev/contributor/continuous_integration.html#skipping

Note that we are a team of volunteers; we appreciate your
patience during the review process.

Again, thanks for contributing!
-->

#### Reference issue
<!--Example: Closes gh-WXYZ.-->
towards https://github.com/scipy/scipy/issues/18279
closes https://github.com/scipy/scipy/issues/17572
#### What does this implement/fix?
<!--Please explain your changes.-->

Before:
```
In [3]: from scipy.optimize.optimize import MemoizeJac
<ipython-input-3-ad2abe888e13>:1: DeprecationWarning: Please use `MemoizeJac` from the `scipy.optimize` namespace, the `scipy.optimize.optimize` namespace is deprecated.
  from scipy.optimize.optimize import MemoizeJac
```

After:
```
In [1]:  from scipy.optimize.optimize import MemoizeJac
<ipython-input-1-ad2abe888e13>:1: DeprecationWarning: `scipy.optimize.optimize.MemoizeJac` is deprecated along with the `scipy.optimize.optimize` namespace. `scipy.optimize.optimize.MemoizeJac` will be removed in SciPy 1.13.0, and the `scipy.optimize.optimize` namespace will be removed in SciPy 2.0.0.
  from scipy.optimize.optimize import MemoizeJac
```
#### Additional information
<!--Any additional information you think is important.-->


Problem statement:
BUG: Deprecation warning says to use non-existent symbols
### Describe your issue.

There is a deprecation warning for importing `MemoizeJac` that recommends importing it from `scipy.optimize` as a new approach, but I can't import it from `scipy.optimize`.

### Reproducing Code Example

```python
Python 3.8.15 | packaged by conda-forge | (default, Nov 22 2022, 08:46:39) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.33.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import scipy

In [2]: scipy.__version__
Out[2]: '1.9.3'

In [3]: from scipy.optimize.optimize import MemoizeJac
<ipython-input-3-ad2abe888e13>:1: DeprecationWarning: Please use `MemoizeJac` from the `scipy.optimize` namespace, the `scipy.optimize.optimize` namespace is deprecated.
  from scipy.optimize.optimize import MemoizeJac

In [4]: from scipy.optimize import MemoizeJac
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-4-2958dec3cdad> in <cell line: 1>()
----> 1 from scipy.optimize import MemoizeJac

ImportError: cannot import name 'MemoizeJac' from 'scipy.optimize' (/home/moorepants/miniconda/lib/python3.8/site-packages/scipy/optimize/__init__.py)

In [5]: import scipy.optimize

In [6]: scipy.optimize.MemoizeJac
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-6-d0f0a395bf46> in <cell line: 1>()
----> 1 scipy.optimize.MemoizeJac

AttributeError: module 'scipy.optimize' has no attribute 'MemoizeJac'
```
```


### Error message

```shell
See above.
```


### SciPy/NumPy/Python version information

1.9.3 1.23.5 sys.version_info(major=3, minor=8, micro=15, releaselevel='final', serial=0)

Hints from issue discussion:
A relative of https://github.com/scipy/scipy/pull/16532#issuecomment-1173112142 ? 
Yes, looks the same. So this will be fixed in 1.10?
I see that the PR you referenced didn't address this class. So I guess the same fix could be applied.
Would you be willing to send a PR with the fix? It should go to the main branch, once it's merged we can ask the RM about backporting to 1.10.
I guess since the fix is going to be similar to those already made, it should be an uncontroversial merge.
I'll have a look.
Are we sure `MemoizeJac` is supposed to be public? It doesn't have a complete docstring. I'd prefer to fix this by ensuring that this error message doesn't appear for functions/methods that are not in `__all__`, and we really should rename `MemoizeJac` to `_MemoizeJac`.
Oh I see @ev-br that in https://github.com/scipy/scipy/pull/16532#issuecomment-1176752145 you suggested being liberal about adding to the namespace. I'll let you deal with any "trouble" that ensues : )
It was public in the past. I'm reporting this because I'm getting deprecation warnings downstream that we can no longer import in the prior supported way and the deprecation warning gives an alternative that does not work.
This is not about being liberal with the namespace. It's just that these things were available, and now they are not. Worse, they emit deprecation warnings that are actually misleading and that users cannot act on: these warnings recommend using non-existing objects from the main namespace. 

So IMO it's a clear bug in application of our deprecation/removal process. Hence the suggestion to fix the bug. And when it's fixed, we can start a regular deprecation process :-).
I thought I'd update this issue with what we discussed in the PR (https://github.com/scipy/scipy/pull/17575#issuecomment-1346011901): we need to change the warning to an `ImportError`, but `MemoizeJac` will not be made public.

Commit message: STY: fix style errors

Patch to review:
diff --git a/scipy/_lib/tests/test_public_api.py b/scipy/_lib/tests/test_public_api.py
index ca4f3872c8c9..e096d40cea21 100644
--- a/scipy/_lib/tests/test_public_api.py
+++ b/scipy/_lib/tests/test_public_api.py
@@ -317,7 +317,18 @@ def check_importable(module_name):
 
 
 @pytest.mark.parametrize(("module_name", "correct_module"),
-                         [('scipy.stats.biasedurn', None),
+                         [('scipy.optimize.cobyla', None),
+                          ('scipy.optimize.lbfgsb', None),
+                          ('scipy.optimize.linesearch', None),
+                          ('scipy.optimize.minpack', None),
+                          ('scipy.optimize.minpack2', None),
+                          ('scipy.optimize.moduleTNC', None),
+                          ('scipy.optimize.nonlin', None),
+                          ('scipy.optimize.optimize', None),
+                          ('scipy.optimize.slsqp', None),
+                          ('scipy.optimize.tnc', None),
+                          ('scipy.optimize.zeros', None),
+                          ('scipy.stats.biasedurn', None),
                           ('scipy.stats.kde', None),
                           ('scipy.stats.morestats', None),
                           ('scipy.stats.mstats_basic', 'mstats'),
diff --git a/scipy/optimize/_minpack_py.py b/scipy/optimize/_minpack_py.py
index f6a04c9f212a..c1ede415f337 100644
--- a/scipy/optimize/_minpack_py.py
+++ b/scipy/optimize/_minpack_py.py
@@ -17,6 +17,10 @@
 
 error = _minpack.error
 
+# deprecated imports to be removed in SciPy 1.13.0
+from numpy import dot, eye, take  # noqa
+from numpy.linalg import inv  # noqa
+
 __all__ = ['fsolve', 'leastsq', 'fixed_point', 'curve_fit']
 
 
diff --git a/scipy/optimize/_optimize.py b/scipy/optimize/_optimize.py
index edb6c53ac499..1c33978ce515 100644
--- a/scipy/optimize/_optimize.py
+++ b/scipy/optimize/_optimize.py
@@ -41,6 +41,9 @@
 from scipy._lib._util import MapWrapper, check_random_state
 from scipy.optimize._differentiable_functions import ScalarFunction, FD_METHODS
 
+# deprecated imports to be removed in SciPy 1.13.0
+from numpy import asfarray  # noqa
+
 
 # standard status messages of optimizers
 _status_message = {'success': 'Optimization terminated successfully.',
diff --git a/scipy/optimize/_slsqp_py.py b/scipy/optimize/_slsqp_py.py
index e133f3af660d..248b48d9d1ed 100644
--- a/scipy/optimize/_slsqp_py.py
+++ b/scipy/optimize/_slsqp_py.py
@@ -25,6 +25,9 @@
 from ._numdiff import approx_derivative
 from ._constraints import old_bound_to_new, _arr_to_scalar
 
+# deprecated imports to be removed in SciPy 1.13.0
+from numpy import exp, inf  # noqa
+
 
 __docformat__ = "restructuredtext en"
 
diff --git a/scipy/optimize/cobyla.py b/scipy/optimize/cobyla.py
index cd013fe4e3d1..46b55b842297 100644
--- a/scipy/optimize/cobyla.py
+++ b/scipy/optimize/cobyla.py
@@ -2,8 +2,7 @@
 # Use the `scipy.optimize` namespace for importing the functions
 # included below.
 
-import warnings
-from . import _cobyla_py
+from scipy._lib.deprecation import _sub_module_deprecation
 
 
 __all__ = [  # noqa: F822
@@ -19,13 +18,6 @@ def __dir__():
     return __all__
 
 def __getattr__(name):
-    if name not in __all__:
-        raise AttributeError(
-            "scipy.optimize.cobyla is deprecated and has no attribute "
-            f"{name}. Try looking in scipy.optimize instead.")
-
-    warnings.warn(f"Please use `{name}` from the `scipy.optimize` namespace, "
-                  "the `scipy.optimize.cobyla` namespace is deprecated.",
-                  category=DeprecationWarning, stacklevel=2)
-
-    return getattr(_cobyla_py, name)
+    return _sub_module_deprecation(sub_package="optimize", module="cobyla",
+                                   private_module="_cobyla_py", all=__all__,
+                                   attribute=name)
diff --git a/scipy/optimize/lbfgsb.py b/scipy/optimize/lbfgsb.py
index 9b7b1b62e92a..989cca20463c 100644
--- a/scipy/optimize/lbfgsb.py
+++ b/scipy/optimize/lbfgsb.py
@@ -2,8 +2,7 @@
 # Use the `scipy.optimize` namespace for importing the functions
 # included below.
 
-import warnings
-from . import _lbfgsb_py
+from scipy._lib.deprecation import _sub_module_deprecation
 
 
 __all__ = [  # noqa: F822
@@ -25,13 +24,6 @@ def __dir__():
 
 
 def __getattr__(name):
-    if name not in __all__:
-        raise AttributeError(
-            "scipy.optimize.lbfgsb is deprecated and has no attribute "
-            f"{name}. Try looking in scipy.optimize instead.")
-
-    warnings.warn(f"Please use `{name}` from the `scipy.optimize` namespace, "
-                  "the `scipy.optimize.lbfgsb` namespace is deprecated.",
-                  category=DeprecationWarning, stacklevel=2)
-
-    return getattr(_lbfgsb_py, name)
+    return _sub_module_deprecation(sub_package="optimize", module="lbfgsb",
+                                   private_module="_lbfgsb_py", all=__all__,
+                                   attribute=name)
diff --git a/scipy/optimize/linesearch.py b/scipy/optimize/linesearch.py
index 90569303d1b7..846deffd8cc3 100644
--- a/scipy/optimize/linesearch.py
+++ b/scipy/optimize/linesearch.py
@@ -2,8 +2,7 @@
 # Use the `scipy.optimize` namespace for importing the functions
 # included below.
 
-import warnings
-from . import _linesearch
+from scipy._lib.deprecation import _sub_module_deprecation
 
 
 __all__ = [  # noqa: F822
@@ -26,13 +25,6 @@ def __dir__():
 
 
 def __getattr__(name):
-    if name not in __all__:
-        raise AttributeError(
-            "scipy.optimize.linesearch is deprecated and has no attribute "
-            f"{name}. Try looking in scipy.optimize instead.")
-
-    warnings.warn(f"Please use `{name}` from the `scipy.optimize` namespace, "
-                  "the `scipy.optimize.linesearch` namespace is deprecated.",
-                  category=DeprecationWarning, stacklevel=2)
-
-    return getattr(_linesearch, name)
+    return _sub_module_deprecation(sub_package="optimize", module="linesearch",
+                                   private_module="_linesearch", all=__all__,
+                                   attribute=name)
diff --git a/scipy/optimize/minpack.py b/scipy/optimize/minpack.py
index cf15278fdee7..b144b30c9f4f 100644
--- a/scipy/optimize/minpack.py
+++ b/scipy/optimize/minpack.py
@@ -2,8 +2,7 @@
 # Use the `scipy.optimize` namespace for importing the functions
 # included below.
 
-import warnings
-from . import _minpack_py
+from scipy._lib.deprecation import _sub_module_deprecation
 
 
 __all__ = [  # noqa: F822
@@ -48,13 +47,6 @@ def __dir__():
 
 
 def __getattr__(name):
-    if name not in __all__:
-        raise AttributeError(
-            "scipy.optimize.minpack is deprecated and has no attribute "
-            f"{name}. Try looking in scipy.optimize instead.")
-
-    warnings.warn(f"Please use `{name}` from the `scipy.optimize` namespace, "
-                  "the `scipy.optimize.minpack` namespace is deprecated.",
-                  category=DeprecationWarning, stacklevel=2)
-
-    return getattr(_minpack_py, name)
+    return _sub_module_deprecation(sub_package="optimize", module="minpack",
+                                   private_module="_minpack_py", all=__all__,
+                                   attribute=name)
diff --git a/scipy/optimize/minpack2.py b/scipy/optimize/minpack2.py
index e2820966d240..b566927f4f46 100644
--- a/scipy/optimize/minpack2.py
+++ b/scipy/optimize/minpack2.py
@@ -2,8 +2,7 @@
 # Use the `scipy.optimize` namespace for importing the functions
 # included below.
 
-import warnings
-from . import _minpack2
+from scipy._lib.deprecation import _sub_module_deprecation
 
 
 __all__ = [  # noqa: F822
@@ -17,13 +16,6 @@ def __dir__():
 
 
 def __getattr__(name):
-    if name not in __all__:
-        raise AttributeError(
-            "scipy.optimize.minpack2 is deprecated and has no attribute "
-            f"{name}. Try looking in scipy.optimize instead.")
-
-    warnings.warn(f"Please use `{name}` from the `scipy.optimize` namespace, "
-                  "the `scipy.optimize.minpack2` namespace is deprecated.",
-                  category=DeprecationWarning, stacklevel=2)
-
-    return getattr(_minpack2, name)
+    return _sub_module_deprecation(sub_package="optimize", module="minpack2",
+                                   private_module="_minpack2", all=__all__,
+                                   attribute=name)
diff --git a/scipy/optimize/moduleTNC.py b/scipy/optimize/moduleTNC.py
index 42e3ae19d4c1..e682b244b542 100644
--- a/scipy/optimize/moduleTNC.py
+++ b/scipy/optimize/moduleTNC.py
@@ -2,13 +2,11 @@
 # Use the `scipy.optimize` namespace for importing the functions
 # included below.
 
-import warnings
-from . import _moduleTNC
 
+from scipy._lib.deprecation import _sub_module_deprecation
 
-__all__ = [  # noqa: F822
 
-]
+__all__ = []  # noqa: F822
 
 
 def __dir__():
@@ -16,13 +14,6 @@ def __dir__():
 
 
 def __getattr__(name):
-    if name not in __all__:
-        raise AttributeError(
-            "scipy.optimize.moduleTNC is deprecated and has no attribute "
-            f"{name}. Try looking in scipy.optimize instead.")
-
-    warnings.warn(f"Please use `{name}` from the `scipy.optimize` namespace, "
-                  "the `scipy.optimize.moduleTNC` namespace is deprecated.",
-                  category=DeprecationWarning, stacklevel=2)
-
-    return getattr(_moduleTNC, name)
+    return _sub_module_deprecation(sub_package="optimize", module="moduleTNC",
+                                   private_module="_moduleTNC", all=__all__,
+                                   attribute=name)
diff --git a/scipy/optimize/nonlin.py b/scipy/optimize/nonlin.py
index 245d6c2a44b2..02ce31c9f219 100644
--- a/scipy/optimize/nonlin.py
+++ b/scipy/optimize/nonlin.py
@@ -2,8 +2,7 @@
 # Use the `scipy.optimize` namespace for importing the functions
 # included below.
 
-import warnings
-from . import _nonlin
+from scipy._lib.deprecation import _sub_module_deprecation
 
 
 __all__ = [  # noqa: F822
@@ -53,13 +52,6 @@ def __dir__():
 
 
 def __getattr__(name):
-    if name not in __all__:
-        raise AttributeError(
-            "scipy.optimize.nonlin is deprecated and has no attribute "
-            f"{name}. Try looking in scipy.optimize instead.")
-
-    warnings.warn(f"Please use `{name}` from the `scipy.optimize` namespace, "
-                  "the `scipy.optimize.nonlin` namespace is deprecated.",
-                  category=DeprecationWarning, stacklevel=2)
-
-    return getattr(_nonlin, name)
+    return _sub_module_deprecation(sub_package="optimize", module="nonlin",
+                                   private_module="_nonlin", all=__all__,
+                                   attribute=name)
diff --git a/scipy/optimize/optimize.py b/scipy/optimize/optimize.py
index 9740388ae86b..a12ccdc783d2 100644
--- a/scipy/optimize/optimize.py
+++ b/scipy/optimize/optimize.py
@@ -2,8 +2,7 @@
 # Use the `scipy.optimize` namespace for importing the functions
 # included below.
 
-import warnings
-from . import _optimize
+from scipy._lib.deprecation import _sub_module_deprecation
 
 
 __all__ = [  # noqa: F822
@@ -35,11 +34,9 @@
     'fmin_powell',
     'fminbound',
     'golden',
-    'is_array_scalar',
     'line_search',
     'line_search_wolfe1',
     'line_search_wolfe2',
-    'main',
     'rosen',
     'rosen_der',
     'rosen_hess',
@@ -59,13 +56,6 @@ def __dir__():
 
 
 def __getattr__(name):
-    if name not in __all__:
-        raise AttributeError(
-            "scipy.optimize.optimize is deprecated and has no attribute "
-            f"{name}. Try looking in scipy.optimize instead.")
-
-    warnings.warn(f"Please use `{name}` from the `scipy.optimize` namespace, "
-                  "the `scipy.optimize.optimize` namespace is deprecated.",
-                  category=DeprecationWarning, stacklevel=2)
-
-    return getattr(_optimize, name)
+    return _sub_module_deprecation(sub_package="optimize", module="optimize",
+                                   private_module="_optimize", all=__all__,
+                                   attribute=name)
diff --git a/scipy/optimize/slsqp.py b/scipy/optimize/slsqp.py
index ff6ac4158b97..c6cca8822b8b 100644
--- a/scipy/optimize/slsqp.py
+++ b/scipy/optimize/slsqp.py
@@ -2,8 +2,7 @@
 # Use the `scipy.optimize` namespace for importing the functions
 # included below.
 
-import warnings
-from . import _slsqp_py
+from scipy._lib.deprecation import _sub_module_deprecation
 
 
 __all__ = [  # noqa: F822
@@ -34,13 +33,6 @@ def __dir__():
 
 
 def __getattr__(name):
-    if name not in __all__:
-        raise AttributeError(
-            "scipy.optimize.slsqp is deprecated and has no attribute "
-            f"{name}. Try looking in scipy.optimize instead.")
-
-    warnings.warn(f"Please use `{name}` from the `scipy.optimize` namespace, "
-                  "the `scipy.optimize.slsqp` namespace is deprecated.",
-                  category=DeprecationWarning, stacklevel=2)
-
-    return getattr(_slsqp_py, name)
+    return _sub_module_deprecation(sub_package="optimize", module="slsqp",
+                                   private_module="_slsqp_py", all=__all__,
+                                   attribute=name)
diff --git a/scipy/optimize/tnc.py b/scipy/optimize/tnc.py
index 9188ddb0b56b..534dd8d68999 100644
--- a/scipy/optimize/tnc.py
+++ b/scipy/optimize/tnc.py
@@ -2,8 +2,7 @@
 # Use the `scipy.optimize` namespace for importing the functions
 # included below.
 
-import warnings
-from . import _tnc
+from scipy._lib.deprecation import _sub_module_deprecation
 
 
 __all__ = [  # noqa: F822
@@ -41,13 +40,6 @@ def __dir__():
 
 
 def __getattr__(name):
-    if name not in __all__:
-        raise AttributeError(
-            "scipy.optimize.tnc is deprecated and has no attribute "
-            f"{name}. Try looking in scipy.optimize instead.")
-
-    warnings.warn(f"Please use `{name}` from the `scipy.optimize` namespace, "
-                  "the `scipy.optimize.tnc` namespace is deprecated.",
-                  category=DeprecationWarning, stacklevel=2)
-
-    return getattr(_tnc, name)
+    return _sub_module_deprecation(sub_package="optimize", module="tnc",
+                                   private_module="_tnc", all=__all__,
+                                   attribute=name)
diff --git a/scipy/optimize/zeros.py b/scipy/optimize/zeros.py
index 239265a195d3..00941c86b9c2 100644
--- a/scipy/optimize/zeros.py
+++ b/scipy/optimize/zeros.py
@@ -2,8 +2,7 @@
 # Use the `scipy.optimize` namespace for importing the functions
 # included below.
 
-import warnings
-from . import _zeros_py
+from scipy._lib.deprecation import _sub_module_deprecation
 
 
 __all__ = [  # noqa: F822
@@ -32,13 +31,6 @@ def __dir__():
 
 
 def __getattr__(name):
-    if name not in __all__:
-        raise AttributeError(
-            "scipy.optimize.zeros is deprecated and has no attribute "
-            f"{name}. Try looking in scipy.optimize instead.")
-
-    warnings.warn(f"Please use `{name}` from the `scipy.optimize` namespace, "
-                  "the `scipy.optimize.zeros` namespace is deprecated.",
-                  category=DeprecationWarning, stacklevel=2)
-
-    return getattr(_zeros_py, name)
+    return _sub_module_deprecation(sub_package="optimize", module="zeros",
+                                   private_module="_zeros_py", all=__all__,
+                                   attribute=name)
\ No newline at end of file


Code review comments:
diff_hunk: @@ -32,13 +31,6 @@ def __dir__():
 
 
 def __getattr__(name):
-    if name not in __all__:
-        raise AttributeError(
-            "scipy.optimize.zeros is deprecated and has no attribute "
-            f"{name}. Try looking in scipy.optimize instead.")
-
-    warnings.warn(f"Please use `{name}` from the `scipy.optimize` namespace, "
-                  "the `scipy.optimize.zeros` namespace is deprecated.",
-                  category=DeprecationWarning, stacklevel=2)
-
-    return getattr(_zeros_py, name)
+    return _sub_module_deprecation(sub_package="optimize", module="zeros",
+                                   private_module="_zeros_py", all=__all__,
+                                   attribute=name)
path: scipy/optimize/zeros.py
text: @user1:
```suggestion
                                   attribute=name)

```