Repository: scipy/scipy

Issue:
it's not possible to tell the `optimize.least_squares` solver to never terminate on xtol
Other trf solvers I've used support this by allowing the user to set xtol=0


Discussion:
``xtol=None`` would be more idiomatic, but other than that sounds reasonable. @nmayorov ?
Our lower bound is machine epsilon for each of the tolerance. For `xtol` it means that we should distinguish between nearby points using double precision numbers.  It seems to me is that `x` didn't change (in floating point numbers) after an iteration then it makes sense to terminate the algorithm.

@greysAcademicCode does it have any benefits to disable it? 
@nmayorov, say `x` is a five element vector, and I have `xtol = np.finfo(float).eps`. Does the algorithm terminate when any element of `x` stops changing or when all elements stop?
If one of the fit parameters can trigger premature termination, I'd say that's a bad thing.

In general though, when I'm working on getting a good fit for a new problem, I like to probe how the fit goes when I change the termination conditions. I learn a lot about the system experimenting that way. That's really why I'd like the option of switching on and off each of the termination conditions here (xtol, gtol, ftol, max_nfev).
The exact condition is `norm(dx) < xtol * (xtol + norm(x))` and `norm` is Euclidean norm. The ideas behind these limits were a) to ensure that the algorithm always terminates b) guide users to realistic tolerances. 

I believe disabling some of the checks is not a bad thing, but I don't know what is "the best" approach here. I think the current approach is fine, but we can consider changing it.
So @greyltc do you really demand this change?
__Demand__ is certainly not the word I'd use.

I would find it _very_ helpful (and I believe others would too) if there was a way to disable the xtol, gtol and ftol termination conditions individually, for example by setting them to zero. Maybe a warning could be issued in that case.
OK, I understand you. Would you send a pull request for that? I think a warning for each tolerance that "setting < tol > below `eps` effectively disables the check" (something along those lines) would be sufficient.
Hello, I'm interested in disabling xtol too. Is there a way to do it yet? Thanks!
I have an issue with xtol in that it uses the norm of x to decide tolerances for all elements (at least in TRF it does). If one of the parameters is much larger than the others, then the fitter can fail to make progress as the largest parameter sets the scaling.

If I could disable it, that would be great. Otherwise perhaps an additional option flag or a callback to decide how best to compute it for the given problem. There could be standard callbacks for the current method, one for the maximum relative change of all parameters, and a callback which disables the computation.
  
@mccullerlp this is a valid point. Now as I look at it I don't understand why exactly I decided to use unscaled `x` for tolerance determination. MINPACK uses scaled `x` and it makes more sense. I will think about it. Maybe we should change the criteria first.
Was wondering if there has been a decision on making `xtol=None` (along with maybe the other termination conditions) available?
@nmayorov is there any progress or branch active regarding this issue?

+1 for `xtol=None` and/or scaled criteria

Resolving patch:
diff --git a/scipy/optimize/_lsq/common.py b/scipy/optimize/_lsq/common.py
--- a/scipy/optimize/_lsq/common.py
+++ b/scipy/optimize/_lsq/common.py
@@ -230,11 +230,12 @@ def update_tr_radius(Delta, actual_reduction, predicted_reduction,
     Delta : float
         New radius.
     ratio : float
-        Ratio between actual and predicted reductions. Zero if predicted
-        reduction is zero.
+        Ratio between actual and predicted reductions.
     """
     if predicted_reduction > 0:
         ratio = actual_reduction / predicted_reduction
+    elif predicted_reduction == actual_reduction == 0:
+        ratio = 1
     else:
         ratio = 0
 
diff --git a/scipy/optimize/_lsq/least_squares.py b/scipy/optimize/_lsq/least_squares.py
--- a/scipy/optimize/_lsq/least_squares.py
+++ b/scipy/optimize/_lsq/least_squares.py
@@ -106,16 +106,22 @@ def prepare_bounds(bounds, n):
 
 
 def check_tolerance(ftol, xtol, gtol):
-    message = "{} is too low, setting to machine epsilon {}."
-    if ftol < EPS:
-        warn(message.format("`ftol`", EPS))
-        ftol = EPS
-    if xtol < EPS:
-        warn(message.format("`xtol`", EPS))
-        xtol = EPS
-    if gtol < EPS:
-        warn(message.format("`gtol`", EPS))
-        gtol = EPS
+    def check(tol, name):
+        if tol is None:
+            tol = 0
+        elif tol < EPS:
+            warn("Setting `{}` below the machine epsilon ({:.2e}) effectively "
+                 "disables the corresponding termination condition."
+                 .format(name, EPS))
+        return tol
+
+    ftol = check(ftol, "ftol")
+    xtol = check(xtol, "xtol")
+    gtol = check(gtol, "gtol")
+
+    if ftol < EPS and xtol < EPS and gtol < EPS:
+        raise ValueError("At least one of the tolerances must be higher than "
+                         "machine epsilon ({:.2e}).".format(EPS))
 
     return ftol, xtol, gtol
 
@@ -293,12 +299,13 @@ def least_squares(
               efficient method for small unconstrained problems.
 
         Default is 'trf'. See Notes for more information.
-    ftol : float, optional
+    ftol : float or None, optional
         Tolerance for termination by the change of the cost function. Default
         is 1e-8. The optimization process is stopped when  ``dF < ftol * F``,
         and there was an adequate agreement between a local quadratic model and
-        the true model in the last step.
-    xtol : float, optional
+        the true model in the last step. If None, the termination by this
+        condition is disabled.
+    xtol : float or None, optional
         Tolerance for termination by the change of the independent variables.
         Default is 1e-8. The exact condition depends on the `method` used:
 
@@ -307,7 +314,8 @@ def least_squares(
               a trust-region radius and ``xs`` is the value of ``x``
               scaled according to `x_scale` parameter (see below).
 
-    gtol : float, optional
+        If None, the termination by this condition is disabled.
+    gtol : float or None, optional
         Tolerance for termination by the norm of the gradient. Default is 1e-8.
         The exact condition depends on a `method` used:
 
@@ -321,6 +329,7 @@ def least_squares(
               between columns of the Jacobian and the residual vector is less
               than `gtol`, or the residual vector is zero.
 
+        If None, the termination by this condition is disabled.
     x_scale : array_like or 'jac', optional
         Characteristic scale of each variable. Setting `x_scale` is equivalent
         to reformulating the problem in scaled variables ``xs = x / x_scale``.
diff --git a/scipy/optimize/_lsq/trf.py b/scipy/optimize/_lsq/trf.py
--- a/scipy/optimize/_lsq/trf.py
+++ b/scipy/optimize/_lsq/trf.py
@@ -358,19 +358,18 @@ def trf_bounds(fun, jac, x0, f0, J0, lb, ub, ftol, xtol, gtol, max_nfev,
             correction = 0.5 * np.dot(step_h * diag_h, step_h)
 
             Delta_new, ratio = update_tr_radius(
-                Delta, actual_reduction - correction, predicted_reduction,
-                step_h_norm, step_h_norm > 0.95 * Delta
-            )
-            alpha *= Delta / Delta_new
-            Delta = Delta_new
+                Delta, actual_reduction, predicted_reduction,
+                step_h_norm, step_h_norm > 0.95 * Delta)
 
             step_norm = norm(step)
             termination_status = check_termination(
                 actual_reduction, cost, step_norm, norm(x), ratio, ftol, xtol)
-
             if termination_status is not None:
                 break
 
+            alpha *= Delta / Delta_new
+            Delta = Delta_new
+
         if actual_reduction > 0:
             x = x_new
 
@@ -523,16 +522,16 @@ def trf_no_bounds(fun, jac, x0, f0, J0, ftol, xtol, gtol, max_nfev,
             Delta_new, ratio = update_tr_radius(
                 Delta, actual_reduction, predicted_reduction,
                 step_h_norm, step_h_norm > 0.95 * Delta)
-            alpha *= Delta / Delta_new
-            Delta = Delta_new
 
             step_norm = norm(step)
             termination_status = check_termination(
                 actual_reduction, cost, step_norm, norm(x), ratio, ftol, xtol)
-
             if termination_status is not None:
                 break
 
+            alpha *= Delta / Delta_new
+            Delta = Delta_new
+
         if actual_reduction > 0:
             x = x_new
 


Test patch:
diff --git a/scipy/optimize/tests/test_least_squares.py b/scipy/optimize/tests/test_least_squares.py
--- a/scipy/optimize/tests/test_least_squares.py
+++ b/scipy/optimize/tests/test_least_squares.py
@@ -367,6 +367,22 @@ def test_bvp(self):
         assert_(res.nfev < max_nfev)
         assert_(res.cost < 0.5)
 
+    def test_error_raised_when_all_tolerances_below_eps(self):
+        # Test that all 0 tolerances are not allowed.
+        assert_raises(ValueError, least_squares, fun_trivial, 2.0,
+                      method=self.method, ftol=None, xtol=None, gtol=None)
+
+    def test_convergence_with_only_one_tolerance_enabled(self):
+        x0 = [-2, 1]
+        x_opt = [1, 1]
+        for ftol, xtol, gtol in [(1e-8, None, None),
+                                  (None, 1e-8, None),
+                                  (None, None, 1e-8)]:
+            res = least_squares(fun_rosenbrock, x0, jac=jac_rosenbrock,
+                                ftol=ftol, gtol=gtol, xtol=xtol,
+                                method=self.method)
+            assert_allclose(res.x, x_opt)
+
 
 class BoundsMixin(object):
     def test_inconsistent(self):
@@ -732,4 +748,3 @@ def test_basic():
     # test that 'method' arg is really optional
     res = least_squares(fun_trivial, 2.0)
     assert_allclose(res.x, 0, atol=1e-10)
-