Repository: scipy/scipy

Issue:
BUG: scipy.optimize.newton says the root of x^2+1 is zero.
Scipy version 1.1.0, Python 3.6.3

```python
>>> def f(x):
...     return x*x+1
...
>>> def f_p(x):
...     return 2*x
>>> newton(f, 1, f_p)
../../scipy/optimize/zeros.py:169: RuntimeWarning: derivative was zero.
  warnings.warn(msg, RuntimeWarning)
0.0
```

See related discussion on [scicomp.stackexchange](https://scicomp.stackexchange.com/questions/30597/defining-a-condition-number-and-termination-criteria-for-newtons-method).


Discussion:
It will work if you use the appropriate  data type. Try newton(f,10 +10j). It will give an answer (1.0782266972886038e-22+1j) which is correct. 
@mrkalling : The point is that if you don't know if a real root exists or not, then you'd like to pass some function in with a real data, and you'd like the error message to be "can't find root" or "root does not exist", not the wrong answer.
`newton` should raise an exception in this case.
So the desired change is that a final check should be done in a root solver (newton) by inserting the final value into the function and check it returns 0?
Starting to browse into this. It looks like @mrkalling suggestion [won't work](https://github.com/scipy/scipy/blob/master/scipy/optimize/zeros.py#L359) if you want to look at vectorized root finder with complex roots, but it will work with a single complex root. @mikofski, did you look at implementing `_array_newton` with complex roots?
I'm looking at a final check of the roots to see that they're actually roots, and there looks to be a lot of branches to take care of in `optimize.newton`.
If `newton` hits a point where the derivative is zero and therefore the algorithm can't continue, it should not trigger a warning and return its current position.  It should raise an exception--the algorithm failed.

I suppose that before raising the exception, it could first check to see if it got lucky and it just happened to be at a point that meets the conditions for a root.  If so, return the root--no warning necessary.  I was going to say "But how likely is that to happen, really?", but then I tried the following example, where there is a root with multiplicity 2 at x=1:

	In [103]: def f(x):
	     ...:     return (x-1)**2
	     ...: 

	In [104]: def f_p(x):
	     ...:     return 2*(x-1)
	     ...: 

	In [105]: newton(f, 0, f_p, tol=1e-20, maxiter=1000)
	/Users/warren/miniconda36b/lib/python3.6/site-packages/scipy/optimize/zeros.py:169: RuntimeWarning: derivative was zero.
	  warnings.warn(msg, RuntimeWarning)
	Out[105]: 1.0

In that case, the returned value is correct.  And I would say, therefore, that the warning is spurious and should not have been generated.  (What I *expected* to happen here is that the algorithm would just converge slowly to the root.  I didn't expect to so easily find an example where the algorithm lands directly on the multiple root.)

So yeah, before giving up and raising an exception when a zero derivative is encountered, check the current point.  If it meets the convergence criteria, return it (and don't generate a warning).  Otherwise, raise an exception.

Does that sound reasonable?

@WarrenWeckesser : Zero derivatives technically are fatal to Newton's method, but really nobody cares about Newton's method, they want a root. So generally algorithms will degrade to bisection when a zero derivative is encountered rather than throwing an exception. This tends to work well in practice.
> So the desired change is that a final check should be done in a root solver (newton) by inserting the final value into the function and check it returns 0?

In fact I thought this was a reasonable behavior, but take a look [here](https://math.stackexchange.com/questions/3017766/constructing-newton-iteration-converging-to-non-root). The only way to get to a convergent Newton sequence to a non-root is to have a discontinuous derivative. Ok, so do you want to support that or not? If not, then the current *idea* for the termination condition is correct (successive approximations are clustered together). But the problem is that the implementation is broken when a zero derivative is encountered.
Has anyone checked this in SciPy-1.2 yet?
@mikofski: Arghhh... I thought I was using a build from the master branch, but I ended up running the above examples in 1.1.0.  When I rerun @NAThompson's example (f(x) = x**2 + 1) using a newly built version `'1.3.0.dev0+ae43f9d'`, I still get the warning with the incorrect return value of 0.  My later example, with `f(x) = (x - 1)**2`, returns 1.0 with no warning (hooray!).

@NAThompson, what you wrote above sounds like a useful enhancement.  My suggestion is for dealing with the immediate problem in a simple way to avoid returning an incorrect root.  Your suggestion and mine are not mutually exclusive--we can make an immediate fix that raises an exception, and then someone can work on enhancing the root-finding code to be more robust. Of course, if someone is going to work on the enhancement right away and try to get it into the next release, then we can skip my suggestion and go straight to the more robust enhancement.  Either way, for your specific example, the code should eventually detect that it is failing to find a real root and raise an exception.
>you'd like the error message to be "can't find root" or "root does not exist", - @NAThompson

>newton should raise an exception in this case. - @WarrenWeckesser

As of SciPy 1.2, please provide arg `disp=True` to raise an exception instead of warning. See [docs](../blob/ae43f9d033b32096c703eb6a00136f8c63b3ecc0/scipy/optimize/zeros.py#L143)
>If True, raise a `RuntimeError` if the algorithm didn't converge,

>did you look at implementing `_array_newton` with complex roots? - @andyfaff 

Yes, complex roots work with `_array_newton`, there is already a [test](../blob/master/scipy/optimize/tests/test_zeros.py#L482), it's for Halley's, which is the same as Newton but with the addition of the 2nd derivative

>before raising the exception, it could first check to see if it got lucky and it just happened to be at a point that meets the conditions for a root. If so, return the root--no warning necessary. - @WarrenWeckesser

This was implemented already in #8907

>nobody cares about Newton's method, they want a root. So generally algorithms will degrade to bisection - @NAThompson

This is nearly what `brentq` does already. The [documents](https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html#scalar-functions) state that Brent is recommended over Newton
>In general `brentq` is the best choice

and since #8890 if the universal scalar root dispatcher is used with a bracket and no derivatives, Brent is used by default

In my opinion this issue can be closed because it's already covered by `disp=True` but maybe @pvanmulbregt should weigh in
@WarrenWeckesser thanks! Can you please try again with the keyword argument `disp=True` ? Thanks again ☺️
@mikofski: `disp=True` is the default; see https://github.com/scipy/scipy/blob/1a3aa92504020b13236b249a6dca44287b332af9/scipy/optimize/zeros.py#L89

As currently implemented (`scipy.__version__` is `'1.3.0.dev0+ae43f9d'`), the `disp` argument doesn't affect the behavior when a zero derivative is encountered. With @NAThompson's example, I get the warning and an incorrect result whether `disp` is True or False.

That's a bug I guess. I have a PR ready, checking tests now locally

Resolving patch:
diff --git a/scipy/optimize/zeros.py b/scipy/optimize/zeros.py
--- a/scipy/optimize/zeros.py
+++ b/scipy/optimize/zeros.py
@@ -286,7 +286,12 @@ class of similar problems can be solved together.
             fder = fprime(p0, *args)
             funcalls += 1
             if fder == 0:
-                msg = "derivative was zero."
+                msg = "Derivative was zero."
+                if disp:
+                    msg += (
+                        " Failed to converge after %d iterations, value is %s."
+                        % (itr + 1, p0))
+                    raise RuntimeError(msg)
                 warnings.warn(msg, RuntimeWarning)
                 return _results_select(
                     full_output, (p0, funcalls, itr + 1, _ECONVERR))
@@ -327,7 +332,12 @@ class of similar problems can be solved together.
         for itr in range(maxiter):
             if q1 == q0:
                 if p1 != p0:
-                    msg = "Tolerance of %s reached" % (p1 - p0)
+                    msg = "Tolerance of %s reached." % (p1 - p0)
+                    if disp:
+                        msg += (
+                            " Failed to converge after %d iterations, value is %s."
+                            % (itr + 1, p1))
+                        raise RuntimeError(msg)
                     warnings.warn(msg, RuntimeWarning)
                 p = (p1 + p0) / 2.0
                 return _results_select(
@@ -346,7 +356,8 @@ class of similar problems can be solved together.
             funcalls += 1
 
     if disp:
-        msg = "Failed to converge after %d iterations, value is %s" % (itr + 1, p)
+        msg = ("Failed to converge after %d iterations, value is %s."
+               % (itr + 1, p))
         raise RuntimeError(msg)
 
     return _results_select(full_output, (p, funcalls, itr + 1, _ECONVERR))


Test patch:
diff --git a/scipy/optimize/tests/test_zeros.py b/scipy/optimize/tests/test_zeros.py
--- a/scipy/optimize/tests/test_zeros.py
+++ b/scipy/optimize/tests/test_zeros.py
@@ -398,14 +398,16 @@ def test_newton_full_output(self):
                 # Check that the correct Exception is raised and
                 # validate the start of the message.
                 with pytest.raises(
-                        RuntimeError,
-                        match='Failed to converge after %d iterations, value is .*' % (iters)):
+                    RuntimeError,
+                    match='Failed to converge after %d iterations, value is .*' % (iters)):
                     x, r = zeros.newton(self.f1, x0, maxiter=iters, disp=True, **kwargs)
 
     def test_deriv_zero_warning(self):
         func = lambda x: x**2 - 2.0
         dfunc = lambda x: 2*x
-        assert_warns(RuntimeWarning, zeros.newton, func, 0.0, dfunc)
+        assert_warns(RuntimeWarning, zeros.newton, func, 0.0, dfunc, disp=False)
+        with pytest.raises(RuntimeError, match='Derivative was zero'):
+            result = zeros.newton(func, 0.0, dfunc)
 
     def test_newton_does_not_modify_x0(self):
         # https://github.com/scipy/scipy/issues/9964
@@ -512,13 +514,17 @@ def test_zero_der_nz_dp():
     p0 = (2.0 - 1e-4) / (2.0 + 1e-4)
     with suppress_warnings() as sup:
         sup.filter(RuntimeWarning, "Tolerance of")
-        x = zeros.newton(lambda y: (y - 1.0) ** 2, x0=p0)
+        x = zeros.newton(lambda y: (y - 1.0) ** 2, x0=p0, disp=False)
     assert_allclose(x, 1)
+    with pytest.raises(RuntimeError, match='Tolerance of'):
+        x = zeros.newton(lambda y: (y - 1.0) ** 2, x0=p0, disp=True)
     p0 = (-2.0 + 1e-4) / (2.0 + 1e-4)
     with suppress_warnings() as sup:
         sup.filter(RuntimeWarning, "Tolerance of")
-        x = zeros.newton(lambda y: (y + 1.0) ** 2, x0=p0)
+        x = zeros.newton(lambda y: (y + 1.0) ** 2, x0=p0, disp=False)
     assert_allclose(x, -1)
+    with pytest.raises(RuntimeError, match='Tolerance of'):
+        x = zeros.newton(lambda y: (y + 1.0) ** 2, x0=p0, disp=True)
 
 
 def test_array_newton_failures():
@@ -682,3 +688,21 @@ def test_gh9254_flag_if_maxiter_exceeded(maximum_iterations, flag_expected):
     elif flag_expected == zeros.CONVERGED:
         # converged before maximum iterations
         assert result[1].iterations < maximum_iterations
+
+
+def test_gh9551_raise_error_if_disp_true():
+    """Test that if disp is true then zero derivative raises RuntimeError"""
+
+    def f(x):
+        return x*x + 1
+
+    def f_p(x):
+        return 2*x
+
+    assert_warns(RuntimeWarning, zeros.newton, f, 1.0, f_p, disp=False)
+    with pytest.raises(
+        RuntimeError,
+        match=r'^Derivative was zero\. Failed to converge after \d+ iterations, value is [+-]?\d*\.\d+\.$'):
+        result = zeros.newton(f, 1.0, f_p)
+    root = zeros.newton(f, complex(10.0, 10.0), f_p)
+    assert_allclose(root, complex(0.0, 1.0))