Repository: scipy/scipy

Issue:
BUG: Iteration index initialisation is wrong in scipy.optimize.linesearch.scalar_search_wolfe2
My issue is about the initialisation of the iteration index in the [line search function which produces a step size satisfying the strong Wolfe conditions (`scipy.optimize.linesearch.scalar_search_wolfe2`)](https://github.com/scipy/scipy/blob/v1.4.1/scipy/optimize/linesearch.py#L411).

#### Reproducing code example:
The algorithm comes from page 60 of [Wright and Nocedal, 'Numerical Optimization', 1999](https://pages.mtu.edu/~struther/Courses/OLD/Sp2013/5630/Jorge_Nocedal_Numerical_optimization_267490.pdf).

The index should be initialised a `i=1` but in scipy it is `for i in xrange(...)` which starts at 0. This has an impact on the chosen path the algorithm takes. Specifically, the condition on line 430 sometimes wont be True and an alternative route will be chosen,

``` python
if (phi_a1 > phi0 + c1 * alpha1 * derphi0) or \
           ((phi_a1 >= phi_a0) and (i > 1)):
    ...
```

I noticed this when implementing the line search for JAX where I use scipy's implementation for a unittest. I was inspecting why I woud sometimes get a difference. The purpose of the `i>1` part is so that that statement is true only when at least one iteration has occured. In scipy's implementation, it implies that it's true only when at least two iterations have occured. This might cause extra function evaluations, or simply make the line search fail in worse case. Quite a simple fix though.

#### Scipy/Numpy/Python version information:
```
1.4.1 1.18.4 sys.version_info(major=3, minor=6, micro=10, releaselevel='final', serial=0)
```

BUG: Iteration index initialisation is wrong in scipy.optimize.linesearch.scalar_search_wolfe2
My issue is about the initialisation of the iteration index in the [line search function which produces a step size satisfying the strong Wolfe conditions (`scipy.optimize.linesearch.scalar_search_wolfe2`)](https://github.com/scipy/scipy/blob/v1.4.1/scipy/optimize/linesearch.py#L411).

#### Reproducing code example:
The algorithm comes from page 60 of [Wright and Nocedal, 'Numerical Optimization', 1999](https://pages.mtu.edu/~struther/Courses/OLD/Sp2013/5630/Jorge_Nocedal_Numerical_optimization_267490.pdf).

The index should be initialised a `i=1` but in scipy it is `for i in xrange(...)` which starts at 0. This has an impact on the chosen path the algorithm takes. Specifically, the condition on line 430 sometimes wont be True and an alternative route will be chosen,

``` python
if (phi_a1 > phi0 + c1 * alpha1 * derphi0) or \
           ((phi_a1 >= phi_a0) and (i > 1)):
    ...
```

I noticed this when implementing the line search for JAX where I use scipy's implementation for a unittest. I was inspecting why I woud sometimes get a difference. The purpose of the `i>1` part is so that that statement is true only when at least one iteration has occured. In scipy's implementation, it implies that it's true only when at least two iterations have occured. This might cause extra function evaluations, or simply make the line search fail in worse case. Quite a simple fix though.

#### Scipy/Numpy/Python version information:
```
1.4.1 1.18.4 sys.version_info(major=3, minor=6, micro=10, releaselevel='final', serial=0)
```

BUG: Iteration index initialisation is wrong in scipy.optimize.linesearch.scalar_search_wolfe2
My issue is about the initialisation of the iteration index in the [line search function which produces a step size satisfying the strong Wolfe conditions (`scipy.optimize.linesearch.scalar_search_wolfe2`)](https://github.com/scipy/scipy/blob/v1.4.1/scipy/optimize/linesearch.py#L411).

#### Reproducing code example:
The algorithm comes from page 60 of [Wright and Nocedal, 'Numerical Optimization', 1999](https://pages.mtu.edu/~struther/Courses/OLD/Sp2013/5630/Jorge_Nocedal_Numerical_optimization_267490.pdf).

The index should be initialised a `i=1` but in scipy it is `for i in xrange(...)` which starts at 0. This has an impact on the chosen path the algorithm takes. Specifically, the condition on line 430 sometimes wont be True and an alternative route will be chosen,

``` python
if (phi_a1 > phi0 + c1 * alpha1 * derphi0) or \
           ((phi_a1 >= phi_a0) and (i > 1)):
    ...
```

I noticed this when implementing the line search for JAX where I use scipy's implementation for a unittest. I was inspecting why I woud sometimes get a difference. The purpose of the `i>1` part is so that that statement is true only when at least one iteration has occured. In scipy's implementation, it implies that it's true only when at least two iterations have occured. This might cause extra function evaluations, or simply make the line search fail in worse case. Quite a simple fix though.

#### Scipy/Numpy/Python version information:
```
1.4.1 1.18.4 sys.version_info(major=3, minor=6, micro=10, releaselevel='final', serial=0)
```



Discussion:
can i work on this?
@pedrooa let me know if you need any explanation from me
can i work on this?
@pedrooa let me know if you need any explanation from me
can i work on this?
@pedrooa let me know if you need any explanation from me

Resolving patch:
diff --git a/scipy/optimize/linesearch.py b/scipy/optimize/linesearch.py
--- a/scipy/optimize/linesearch.py
+++ b/scipy/optimize/linesearch.py
@@ -437,8 +437,9 @@ def scalar_search_wolfe2(phi, derphi, phi0=None,
             warn(msg, LineSearchWarning)
             break
 
+        not_first_iteration = i > 0
         if (phi_a1 > phi0 + c1 * alpha1 * derphi0) or \
-           ((phi_a1 >= phi_a0) and (i > 1)):
+           ((phi_a1 >= phi_a0) and not_first_iteration):
             alpha_star, phi_star, derphi_star = \
                         _zoom(alpha0, alpha1, phi_a0,
                               phi_a1, derphi_a0, phi, derphi,


Test patch:
diff --git a/scipy/optimize/tests/test_linesearch.py b/scipy/optimize/tests/test_linesearch.py
--- a/scipy/optimize/tests/test_linesearch.py
+++ b/scipy/optimize/tests/test_linesearch.py
@@ -163,6 +163,26 @@ def derphi(alpha):
                                   ls.scalar_search_wolfe2, phi, derphi, amax=0.001)
         assert_(s is None)
 
+    def test_scalar_search_wolfe2_regression(self):
+        # Regression test for gh-12157
+        # This phi has its minimum at alpha=4/3 ~ 1.333.
+        def phi(alpha):
+            if alpha < 1:
+                return - 3*np.pi/2 * (alpha - 1)
+            else:
+                return np.cos(3*np.pi/2 * alpha - np.pi)
+
+        def derphi(alpha):
+            if alpha < 1:
+                return - 3*np.pi/2
+            else:
+                return - 3*np.pi/2 * np.sin(3*np.pi/2 * alpha - np.pi)
+
+        s, _, _, _ = ls.scalar_search_wolfe2(phi, derphi)
+        # Without the fix in gh-13073, the scalar_search_wolfe2
+        # returned s=2.0 instead.
+        assert_(s < 1.5)
+
     def test_scalar_search_armijo(self):
         for name, phi, derphi, old_phi0 in self.scalar_iter():
             s, phi1 = ls.scalar_search_armijo(phi, phi(0), derphi(0))