Repository: scipy/scipy

Issue:
Add wrapper for ?gttrf/?gttrs
See [NumFOCUS Small Development Grant Proposal 2019 Round 3 proposal](https://www.dropbox.com/s/i24h3j2osol56uf/Small%20Development%20Grant%20-%20LAPACK%20%20%28Condensed%29.pdf?dl=0) for background.

Add wrapper for [?gttrf](http://www.netlib.org/lapack/explore-html/d1/d40/group__double_g_tcomputational_ga8ca64e542924cec56cbe9837b77d25b7.html)/[?gttrs](http://www.netlib.org/lapack/explore-html/d1/d40/group__double_g_tcomputational_ga887f9fe1a0c6a4aeac1752da5652014a.html).
(This looks like a pretty straightforward way to start.)

Existing [wrapper](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.lapack.dgtsv.html#scipy.linalg.lapack.dgtsv) for [?gtsv](https://www.netlib.org/lapack/explore-html/d4/d62/group__double_g_tsolve_ga2bf93f2ddefa5e671866eb2191dc19d4.html) provides some guidance.

Suggested signatures:
```
! dl, d, du, du2, ipiv, info  = gttrf(dl, d, du, overwrite_dl=0, overwrite_d=0, overwrite_du=0)
```
```
! x, info  = gttrs(dl, d, du, du2, ipiv, b, trans="N", overwrite_b=0)
```

Test idea:

- Generate random matrix `A`
- Decompose `A` with `?gttrf`
- Multiply factors from `?gttrf` and compare with original `A`
- Generate random solutions `x` _*Update: should test for multiple RHS*_
- Generate `b` from `A@x`
- Solve using `?gttrs`
- Compare solution from `?gttrs` against known `x`

Be sure to perform similar tests for the different `trans` options.
_Update: Also test for singular matrix._
_Update: Also test for incorrect/incompatible array sizes._

Also: implement all examples from the [NAG manual](https://www.nag.com/numeric/fl/nagdoc_latest/html/f07/f07conts.html) as additional tests.








Discussion:
Damn. This means I need to revisit `syevr`/`heevr` stuff very soon  😅 
@ilayn we'd certainly appreciate your thoughts on that, but you'll have help! I'd appreciate your thoughts on the proposed signatures and tests for these, if you have a minute. For instance, I thought about renaming the outputs of `?gttrf` (e.g. `dl` -> `ll` for the subdiagonal of the lower triangular factor) but decided against it based on the names of inputs to `?gttrf` in the LAPACK documentation.
The actual variable names are not so important since they have historically very little intuitiveness hence nobody would really care but following LAPACK official docs makes it helpful for newcomers in case of finding help outside SciPy circles. 

What I mean with syevr/heevr is that currently we have an MKL `lwork` bug and our old (current) wrapper intertwined hence some backwards compatibility issues await :) (always fun).
Regarding the tests, could we avoid generating random matrices? Otherwise, those all look good.  
Regarding the signatures I would prefer to be consistent with the LAPACK documentation.

I've created the branch ``lapack_support_gttrf_gttrs`` to which your student may add the tests once we are happy with them.
@Kai-Striega what's wrong with (psuedo)random matrices? Of course we'd fix the seed for repeatability.
Random matrices test one specific case, however, to me, unit tests verify some *specific* functionality with a clear pass or fail condition. e.g. does the function handle over/underflow correctly, does it overwrite the data, does it handle invalid values as we want it to?

E.g. there was a bug in numpy-financial due to a [denormal number](https://en.wikipedia.org/wiki/Denormal_number) being passed. Could that have been caught from a random matrix? Maybe. But I would prefer to have an extra case where we know that the expected outcome occurs.
I think we agree on the philosophy of unit tests. Let's make sure our goals for the unit tests for this issue are the same:

- We are testing for issues specific to the signature of the f2py wrappers for `?gttrf` and `?gttrs`.
- We are not testing the f2py system itself / checking for issues that might be shared with the wrappers for other routines.
- We are not testing the LAPACK routines `?gttrf` or `?gttrs` themselves.
- We are not testing for behavior in case of input type or value errors, as this is a relatively low-level interface, so the wrapper itself doesn't guarantee nice behavior in these cases. Code for handling that sort of thing would go in a higher-level function like `scipy.linalg.lu` or similar, so I don't think we need to write tests with those sorts of inputs here.

So the specific functionality I am testing is that Python and Fortran are talking to one another correctly, and the clear pass criterion is whether the result is correct for a typical, rather than pathological, test case. It might be nice to see what it does with a singular matrix, too, but IMO it could still be a random singular matrix.

Do the tests already in [test_lapack.py](https://github.com/scipy/scipy/blob/master/scipy/linalg/tests/test_lapack.py) go much further than this? 

> Random matrices test one specific case, however, to me, unit tests verify some specific functionality with a clear pass or fail condition. e.g. does the function handle over/underflow correctly, does it overwrite the data, does it handle invalid values as we want it to?

Same applies to the other way around. Sometimes deterministic tests miss the problems that only random big/small size tests can catch say workspace size or cache segfault limitations. Best is having all but trades off with the effort goes in. Hence there is nothing essentially wrong with random matrices but if you already know some corner cases the better it is.
Thanks @ilayn. I agree with that, too. 
I don't know any corner cases that are within my goals. Do you, @Kai-Striega? I could probably come up with corner cases designed to test, for example, the underlying LAPACK routine, but I don't know enough about how f2py works to design corner cases specific to our wrapper.
The tests should generally cover basic functionality. Does it work with correct sizes? Is the result correct? If there are extra options, say, transpose etc. do they actually work? Does it reject wrong sizes? Does it handle empty matrices? etc. etc. All should gracefully error out instead of a segfault which is the common thing when you are trying to debug a mistake somewhere. 

We don't need to test f2py or the LAPACK routine itself 
> Do the tests already in test_lapack.py go much further than this?

There are sometimes checked e.g. test_sytrd checks that a 0x0 array raises an error.  But yes I should have checked the existing tests first. Let's match those for scope plus the edge cases @Ilyan mentioned (rejects wrong sizes, handle empty matrices etc.)

>  Sometimes deterministic tests miss the problems that only random big/small size tests can catch say workspace size or cache segfault limitations.

:+1:  I think having the deterministic tests is just a personal preference.


@mdhaber should the second suggested signature be for ``?gttrs`` not ``?gttrf``?
@mdhaber I've had an attempt at this but I'm getting a strange error I can't seem to figure out. 

```python
>>> dl = np.array([2.1, -1.0, 1.9, 8.0], dtype=double)
>>> d = np.array([3.0, 2.3, -5.0, -0.9, 7.1], dtype=double)
>>> du = np.array([3.4, 3.6, 7.0, -6.0], dtype=double)
>>> gttrf = get_lapack_funcs('gttrf', dtype=dtype)
>>> gttrf(dl, d, du)
E ValueError: failed to create intent(cache|hide)|optional array-- must have defined dimensions but got (-1,)
```
Here's what I've added for the wrapper itself, could you see if there is anything "obvious" I'm missing?

```fortran
subroutine <prefix>gttrf(n, dl, d, du, du2, ipiv, info)
    ! dl, d, du, du2, ipiv, info  = gttrf(dl, d, du)
    ! ?GTTRF computes an LU factorization of a tridiagonal matrix A
    ! using elimination with partial pivoting and row interchanges.
    !
    ! The factorization has the form
    !    A = L * U
    !
    ! where L is a product of permutation and unit lower bidiagonal
    ! matrices and U is upper triangular with nonzeros in only the main
    ! diagonal and first two superdiagonals.
    callstatement (*f2py_func)(&n, dl, d, du, du2, ipiv, &info)
    callprotoargument int*, <ctype>*, <ctype>*, <ctype>*, <ctype>*, int*, int*

    integer intent(hide),   depend(d)                 :: n = len(d)
    <ftype> intent(in,out), depend(n), dimension(n-1) :: dl
    <ftype> intent(in,out), depend(n), dimension(n)   :: d
    <ftype> intent(in,out), depend(n), dimension(n-1) :: du
    <ftype> intent(out),    depend(n), dimension(n-2) :: du2
    integer intent(out),    depend(n), dimension(n)   :: ipiv
    integer intent(out)                               :: info

end subroutine <prefix>gttrf
```
`d` does not depend on `n`. It's the other way around. But in any case this will need more guards against creating negative sized arrays when n is 2 or fewer.
Thanks, that's got it working now! And I am planning to add the checks, but I'd prefer to have a working example first.
@Kai-Striega Does your branch `lapack_support_gttrf_gttrs` have your wrapper yet? It looks like it's only behind master, no commits are ahead of master. Also, I checked it out and tried:
```
gttrf = get_lapack_funcs(('gttrf',), (A[0, 0],))
```
and that failed. Maybe you haven't pushed recently?
It doesn't have anything yet - I'm waiting on the tests to be added.
Ah. OK. I think in the future we can try to write the tests before you write the wrapper. In this case, though, can you add your wrappers first since you've already written them? I would like to work on the tests with @swallan, and it would be nice to be able to check that the test is executable for this first one. 
Sure thing. I'm on my work computer, so I can't push them right now but I'll push them when I get home tonight.

Resolving patch:
diff --git a/scipy/linalg/lapack.py b/scipy/linalg/lapack.py
--- a/scipy/linalg/lapack.py
+++ b/scipy/linalg/lapack.py
@@ -707,6 +707,16 @@
    cgemqrt
    zgemqrt
 
+   sgttrf
+   dgttrf
+   cgttrf
+   zgttrf
+
+   sgttrs
+   dgttrs
+   cgttrs
+   zgttrs
+
    stpqrt
    dtpqrt
    ctpqrt


Test patch:
diff --git a/scipy/linalg/tests/test_lapack.py b/scipy/linalg/tests/test_lapack.py
old mode 100755
new mode 100644
--- a/scipy/linalg/tests/test_lapack.py
+++ b/scipy/linalg/tests/test_lapack.py
@@ -40,6 +40,14 @@
 DTYPES = REAL_DTYPES + COMPLEX_DTYPES
 
 
+def generate_random_dtype_array(shape, dtype):
+    # generates a random matrix of desired data type of shape
+    if dtype in COMPLEX_DTYPES:
+        return (np.random.rand(*shape)
+                + np.random.rand(*shape)*1.0j).astype(dtype)
+    return np.random.rand(*shape).astype(dtype)
+
+
 def test_lapack_documented():
     """Test that all entries are in the doc."""
     if lapack.__doc__ is None:  # just in case there is a python -OO
@@ -1786,6 +1794,161 @@ def test_getc2_gesc2():
                                       x/scale, decimal=4)
 
 
+@pytest.mark.parametrize("dtype", DTYPES)
+def test_gttrf_gttrs(dtype):
+    # The test uses ?gttrf and ?gttrs to solve a random system for each dtype,
+    # tests that the output of ?gttrf define LU matricies, that input
+    # parameters are unmodified, transposal options function correctly, that
+    # incompatible matrix shapes raise an error, and singular matrices return
+    # non zero info.
+
+    seed(42)
+    n = 10
+    atol = 100 * np.finfo(dtype).eps
+
+    # create the matrix in accordance with the data type
+    du = generate_random_dtype_array((n-1,), dtype=dtype)
+    d = generate_random_dtype_array((n,), dtype=dtype)
+    dl = generate_random_dtype_array((n-1,), dtype=dtype)
+
+    diag_cpy = [dl.copy(), d.copy(), du.copy()]
+
+    A = np.diag(d) + np.diag(dl, -1) + np.diag(du, 1)
+    x = np.random.rand(n)
+    b = A @ x
+
+    gttrf, gttrs = get_lapack_funcs(('gttrf', 'gttrs'), dtype=dtype)
+
+    _dl, _d, _du, du2, ipiv, info = gttrf(dl, d, du)
+    # test to assure that the inputs of ?gttrf are unmodified
+    assert_array_equal(dl, diag_cpy[0])
+    assert_array_equal(d, diag_cpy[1])
+    assert_array_equal(du, diag_cpy[2])
+
+    # generate L and U factors from ?gttrf return values
+    # L/U are lower/upper triangular by construction (initially and at end)
+    U = np.diag(_d, 0) + np.diag(_du, 1) + np.diag(du2, 2)
+    L = np.eye(n, dtype=dtype)
+
+    for i, m in enumerate(_dl):
+        # L is given in a factored form.
+        # See http://www.hpcavf.uclan.ac.uk/softwaredoc/sgi_scsl_html/sgi_html/ch03.html
+        piv = ipiv[i] - 1
+        # right multiply by permutation matrix
+        L[:, [i, piv]] = L[:, [piv, i]]
+        # right multiply by Li, rank-one modification of identity
+        L[:, i] += L[:, i+1]*m
+
+    # one last permutation
+    i, piv = -1, ipiv[-1] - 1
+    # right multiply by final permutation matrix
+    L[:, [i, piv]] = L[:, [piv, i]]
+
+    # check that the outputs of ?gttrf define an LU decomposition of A
+    assert_allclose(A, L @ U, atol=atol)
+
+    b_cpy = b.copy()
+    x_gttrs, info = gttrs(_dl, _d, _du, du2, ipiv, b)
+    # test that the inputs of ?gttrs are unmodified
+    assert_array_equal(b, b_cpy)
+    # test that the result of ?gttrs matches the expected input
+    assert_allclose(x, x_gttrs, atol=atol)
+
+    # test that ?gttrf and ?gttrs work with transposal options
+    if dtype in REAL_DTYPES:
+        trans = "T"
+        b_trans = A.T @ x
+    else:
+        trans = "C"
+        b_trans = A.conj().T @ x
+
+    x_gttrs, info = gttrs(_dl, _d, _du, du2, ipiv, b_trans, trans=trans)
+    assert_allclose(x, x_gttrs, atol=atol)
+
+    # test that ValueError is raised with incompatible matrix shapes
+    with assert_raises(ValueError):
+        gttrf(dl[:-1], d, du)
+    with assert_raises(ValueError):
+        gttrf(dl, d[:-1], du)
+    with assert_raises(ValueError):
+        gttrf(dl, d, du[:-1])
+
+    # test that matrix of size n=2 raises exception
+    with assert_raises(Exception):
+        gttrf(dl[0], d[:1], du[0])
+
+    # test that singular (row of all zeroes) matrix fails via info
+    du[0] = 0
+    d[0] = 0
+    __dl, __d, __du, _du2, _ipiv, _info = gttrf(dl, d, du)
+    np.testing.assert_(__d[info - 1] == 0,
+                       "?gttrf: _d[info-1] is {}, not the illegal value :0."
+                       .format(__d[info - 1]))
+
+
+@pytest.mark.parametrize("du, d, dl, du_exp, d_exp, du2_exp, ipiv_exp, b, x",
+                         [(np.array([2.1, -1.0, 1.9, 8.0]),
+                             np.array([3.0, 2.3, -5.0, -.9, 7.1]),
+                             np.array([3.4, 3.6, 7.0, -6.0]),
+                             np.array([2.3, -5, -.9, 7.1]),
+                             np.array([3.4, 3.6, 7, -6, -1.015373]),
+                             np.array([-1, 1.9, 8]),
+                             np.array([2, 3, 4, 5, 5]),
+                             np.array([[2.7, 6.6],
+                                       [-0.5, 10.8],
+                                       [2.6, -3.2],
+                                       [0.6, -11.2],
+                                       [2.7, 19.1]
+                                       ]),
+                             np.array([[-4, 5],
+                                       [7, -4],
+                                       [3, -3],
+                                       [-4, -2],
+                                       [-3, 1]])),
+                          (
+                             np.array([2 - 1j, 2 + 1j, -1 + 1j, 1 - 1j]),
+                             np.array([-1.3 + 1.3j, -1.3 + 1.3j,
+                                       -1.3 + 3.3j, - .3 + 4.3j,
+                                       -3.3 + 1.3j]),
+                             np.array([1 - 2j, 1 + 1j, 2 - 3j, 1 + 1j]),
+                             # du exp
+                             np.array([-1.3 + 1.3j, -1.3 + 3.3j,
+                                       -0.3 + 4.3j, -3.3 + 1.3j]),
+                             np.array([1 - 2j, 1 + 1j, 2 - 3j, 1 + 1j,
+                                       -1.3399 + 0.2875j]),
+                             np.array([2 + 1j, -1 + 1j, 1 - 1j]),
+                             np.array([2, 3, 4, 5, 5]),
+                             np.array([[2.4 - 5j, 2.7 + 6.9j],
+                                       [3.4 + 18.2j, - 6.9 - 5.3j],
+                                       [-14.7 + 9.7j, - 6 - .6j],
+                                       [31.9 - 7.7j, -3.9 + 9.3j],
+                                       [-1 + 1.6j, -3 + 12.2j]]),
+                             np.array([[1 + 1j, 2 - 1j],
+                                       [3 - 1j, 1 + 2j],
+                                       [4 + 5j, -1 + 1j],
+                                       [-1 - 2j, 2 + 1j],
+                                       [1 - 1j, 2 - 2j]])
+                            )])
+def test_gttrf_gttrs_NAG_f07cdf_f07cef_f07crf_f07csf(du, d, dl, du_exp, d_exp,
+                                                     du2_exp, ipiv_exp, b, x):
+    # test to assure that wrapper is consistent with NAG Library Manual Mark 26
+    # example problems: f07cdf and f07cef (real)
+    # examples: f07crf and f07csf (complex)
+    # (Links may expire, so search for "NAG Library Manual Mark 26" online)
+
+    gttrf, gttrs = get_lapack_funcs(('gttrf', "gttrs"), (du[0], du[0]))
+
+    _dl, _d, _du, du2, ipiv, info = gttrf(dl, d, du)
+    assert_allclose(du2, du2_exp)
+    assert_allclose(_du, du_exp)
+    assert_allclose(_d, d_exp, atol=1e-4)  # NAG examples provide 4 decimals.
+    assert_allclose(ipiv, ipiv_exp)
+
+    x_gttrs, info = gttrs(_dl, _d, _du, du2, ipiv, b)
+
+    assert_allclose(x_gttrs, x)
+
+
 @pytest.mark.parametrize('dtype', DTYPES)
 @pytest.mark.parametrize('shape', [(3, 7), (7, 3), (2**18, 2**18)])
 def test_geqrfp_lwork(dtype, shape):
@@ -1795,14 +1958,6 @@ def test_geqrfp_lwork(dtype, shape):
     assert_equal(info, 0)
 
 
-def generate_random_dtype_array(shape, dtype):
-    # generates a random matrix of desired data type of shape
-    if dtype in COMPLEX_DTYPES:
-        return (np.random.rand(*shape)
-                + np.random.rand(*shape)*1.0j).astype(dtype)
-    return np.random.rand(*shape).astype(dtype)
-
-
 @pytest.mark.parametrize('dtype', DTYPES)
 @pytest.mark.parametrize('matrix_size', [(3, 4), (7, 6), (6, 6)])
 def test_geqrfp(dtype, matrix_size):