Repository: scipy/scipy

Issue:
Enhancement: Add LDLt factorization to scipy
It would be useful to be able to use an LDLt decomposition in scipy (similar to http://www.mathworks.com/help/matlab/ref/ldl.html).

This has been discussed and partially developed for numpy in numpy/numpy#3960.
From that discussion it appears a simple wrapper around an LAPACK function for LDLt would suffice, and someone has submitted some code to do that, though no pull request has been submitted.

However (also discussed there) it appears that ideally new linear algebra routines should go into scipy instead of numpy ?

Which has lead to this issue being opened.

Please advise if this sounds appropriate for scipy and if so I am happy to either move @tmwright 's changes over from numpy/numpy#3960 over to scipy (once he gives permission) or start from scratch on a new wrapper. 

Referencing @seberg so he is aware of this new ticket in scipy.



Discussion:
Yes, Scipy sounds like the right place to me. (rationale given in https://github.com/numpy/numpy/issues/3960#issuecomment-223502810)

I noticed a missing LDL decomposition in scipy as well, so I wrote one myself trying to vectorize as much as possible (only 1 for-loop left). :)

The resulting Python function performs a decomposition of a 2000x2000 matrix in 4.56 secs whereas the scipy default cholesky decomposition finishes in 1.61 secs. I know that both algorithms are not 100% comparable but I think it's enough to get an idea of how worse the performance would be.

Does a naive LDL decomposition besides wrappers for LAPACK makes sense for inclusion in scipy? If so, I would start a pull request.

I've naively implemented and tested LDL decomposition, solving, and inversion in [arb_mat](https://github.com/fredrik-johansson/arb/tree/master/arb_mat) in response to https://github.com/fredrik-johansson/arb/pull/110#issuecomment-193654444. `arb` itself is LGPL but I think I could give permission for code derived from the LDL part to be used in a less restrictively licensed project.

A "naïve" implementation makes no sence. It will be slow and error prone. Use LAPACK if you can for `scipy.linalg`. If you cannot find a suitable LAPACK routine, write a solver on top of BLAS. In this case (LDL' and UDU' decompositions) the LAPACK `*SYTRF` routines will do what we need, so preferably use those.

The main coding task will be to parse the output of `*SYTRF`, which is really gross.

@sturlamolden Please see my remarks over on https://github.com/numpy/numpy/issues/3960.

@sturlamolden I would hardly call two lines of code error-prone. ;)

I just thought it would be a good idea to give something back to SciPy as a way to say "thank you".

@tmwright Do you mean both implementations (with and without pivoting) might be useful to have in SciPy?


Resolving patch:
diff --git a/scipy/linalg/__init__.py b/scipy/linalg/__init__.py
--- a/scipy/linalg/__init__.py
+++ b/scipy/linalg/__init__.py
@@ -70,6 +70,7 @@
    svdvals - Singular values of a matrix
    diagsvd - Construct matrix of singular values from output of svd
    orth - Construct orthonormal basis for the range of A using svd
+   ldl - LDL.T decomposition of a Hermitian or a symmetric matrix.
    cholesky - Cholesky decomposition of a matrix
    cholesky_banded - Cholesky decomp. of a sym. or Hermitian banded matrix
    cho_factor - Cholesky decomposition for use in solving a linear system
@@ -187,6 +188,7 @@
 from .basic import *
 from .decomp import *
 from .decomp_lu import *
+from ._decomp_ldl import *
 from .decomp_cholesky import *
 from .decomp_qr import *
 from ._decomp_qz import *
diff --git a/scipy/linalg/_decomp_ldl.py b/scipy/linalg/_decomp_ldl.py
new file mode 100644
--- /dev/null
+++ b/scipy/linalg/_decomp_ldl.py
@@ -0,0 +1,354 @@
+from __future__ import division, print_function, absolute_import
+
+from warnings import warn
+
+import numpy as np
+from numpy import (atleast_2d, ComplexWarning, arange, zeros_like, imag, diag,
+                   iscomplexobj, tril, triu, argsort, empty_like)
+from .decomp import _asarray_validated
+from .lapack import get_lapack_funcs, _compute_lwork
+
+__all__ = ['ldl']
+
+
+def ldl(A, lower=True, hermitian=True, overwrite_a=False, check_finite=True):
+    """ Computes the LDLt or Bunch-Kaufman factorization of a symmetric/
+    hermitian matrix.
+
+    This function returns a block diagonal matrix D consisting blocks of size
+    at most 2x2 and also a possibly permuted unit lower triangular matrix
+    ``L`` such that the factorization ``A = L D L^H`` or ``A = L D L^T``
+    holds. If ``lower`` is False then (again possibly permuted) upper
+    triangular matrices are returned as outer factors.
+
+    The permutation array can be used to triangularize the outer factors
+    simply by a row shuffle, i.e., ``lu[perm, :]`` is an upper/lower
+    triangular matrix. This is also equivalent to multiplication with a
+    permutation matrix ``P.dot(lu)`` where ``P`` is a column-permuted
+    identity matrix ``I[:, perm]``.
+
+    Depending on the value of the boolean ``lower``, only upper or lower
+    triangular part of the input array is referenced. Hence a triangular
+    matrix on entry would give the same result as if the full matrix is
+    supplied.
+
+    Parameters
+    ----------
+    a : array_like
+        Square input array
+    lower : bool, optional
+        This switches between the lower and upper triangular outer factors of
+        the factorization. Lower triangular (``lower=True``) is the default.
+    hermitian : bool, optional
+        For complex-valued arrays, this defines whether ``a = a.conj().T`` or
+        ``a = a.T`` is assumed. For real-valued arrays, this switch has no
+        effect.
+    overwrite_a : bool, optional
+        Allow overwriting data in ``a`` (may enhance performance). The default
+        is False.
+    check_finite : bool, optional
+        Whether to check that the input matrices contain only finite numbers.
+        Disabling may give a performance gain, but may result in problems
+        (crashes, non-termination) if the inputs do contain infinities or NaNs.
+
+    Returns
+    -------
+    lu : ndarray
+        The (possibly) permuted upper/lower triangular outer factor of the
+        factorization.
+    d : ndarray
+        The block diagonal multiplier of the factorization.
+    perm : ndarray
+        The row-permutation index array that brings lu into triangular form.
+
+    Raises
+    ------
+    ValueError
+        If input array is not square.
+    ComplexWarning
+        If a complex-valued array with nonzero imaginary parts on the
+        diagonal is given and hermitian is set to True.
+
+    Examples
+    --------
+    Given an upper triangular array `a` that represents the full symmetric
+    array with its entries, obtain `l`, 'd' and the permutation vector `perm`:
+
+    >>> import numpy as np
+    >>> from scipy.linalg import ldl
+    >>> a = np.array([[2, -1, 3], [0, 2, 0], [0, 0, 1]])
+    >>> lu, d, perm = ldl(a, lower=0) # Use the upper part
+    >>> lu
+    array([[ 0. ,  0. ,  1. ],
+           [ 0. ,  1. , -0.5],
+           [ 1. ,  1. ,  1.5]])
+    >>> d
+    array([[-5. ,  0. ,  0. ],
+           [ 0. ,  1.5,  0. ],
+           [ 0. ,  0. ,  2. ]])
+    >>> perm
+    array([2, 1, 0])
+    >>> lu[perm, :]
+    array([[ 1. ,  1. ,  1.5],
+           [ 0. ,  1. , -0.5],
+           [ 0. ,  0. ,  1. ]])
+    >>> lu.dot(d).dot(lu.T)
+    array([[ 2., -1.,  3.],
+           [-1.,  2.,  0.],
+           [ 3.,  0.,  1.]])
+
+    Notes
+    -----
+    This function uses ``?SYTRF`` routines for symmetric matrices and
+    ``?HETRF`` routines for Hermitian matrices from LAPACK. See [1]_ for
+    the algorithm details.
+
+    Depending on the ``lower`` keyword value, only lower or upper triangular
+    part of the input array is referenced. Moreover, this keyword also defines
+    the structure of the outer factors of the factorization.
+
+    .. versionadded:: 1.1.0
+
+    See also
+    --------
+    cholesky, lu
+
+    References
+    ----------
+    .. [1] J.R. Bunch, L. Kaufman, Some stable methods for calculating
+       inertia and solving symmetric linear systems, Math. Comput. Vol.31,
+       1977. DOI: 10.2307/2005787
+
+    """
+    a = atleast_2d(_asarray_validated(A, check_finite=check_finite))
+    if a.shape[0] != a.shape[1]:
+        raise ValueError('The input array "a" should be square.')
+    # Return empty arrays for empty square input
+    if a.size == 0:
+        return empty_like(a), empty_like(a), np.array([], dtype=int)
+
+    n = a.shape[0]
+    r_or_c = complex if iscomplexobj(a) else float
+
+    # Get the LAPACK routine
+    if r_or_c is complex and hermitian:
+        s, sl = 'hetrf', 'hetrf_lwork'
+        if np.any(imag(diag(a))):
+            warn('scipy.linalg.ldl():\nThe imaginary parts of the diagonal'
+                 'are ignored. Use "hermitian=False" for factorization of'
+                 'complex symmetric arrays.', ComplexWarning, stacklevel=2)
+    else:
+        s, sl = 'sytrf', 'sytrf_lwork'
+
+    solver, solver_lwork = get_lapack_funcs((s, sl), (a,))
+    lwork = _compute_lwork(solver_lwork, n, lower=lower)
+    ldu, piv, info = solver(a, lwork=lwork, lower=lower,
+                            overwrite_a=overwrite_a)
+    if info < 0:
+        raise ValueError('{} exited with the internal error "illegal value '
+                         'in argument number {}". See LAPACK documentation '
+                         'for the error codes.'.format(s.upper(), -info))
+
+    swap_arr, pivot_arr = _ldl_sanitize_ipiv(piv, lower=lower)
+    d, lu = _ldl_get_d_and_l(ldu, pivot_arr, lower=lower, hermitian=hermitian)
+    lu, perm = _ldl_construct_tri_factor(lu, swap_arr, pivot_arr, lower=lower)
+
+    return lu, d, perm
+
+
+def _ldl_sanitize_ipiv(a, lower=True):
+    """
+    This helper function takes the rather strangely encoded permutation array
+    returned by the LAPACK routines ?(HE/SY)TRF and converts it into
+    regularized permutation and diagonal pivot size format.
+
+    Since FORTRAN uses 1-indexing and LAPACK uses different start points for
+    upper and lower formats there are certain offsets in the indices used
+    below.
+
+    Let's assume a result where the matrix is 6x6 and there are two 2x2
+    and two 1x1 blocks reported by the routine. To ease the coding efforts,
+    we still populate a 6-sized array and fill zeros as the following ::
+
+        pivots = [2, 0, 2, 0, 1, 1]
+
+    This denotes a diagonal matrix of the form ::
+
+        [x x        ]
+        [x x        ]
+        [    x x    ]
+        [    x x    ]
+        [        x  ]
+        [          x]
+
+    In other words, we write 2 when the 2x2 block is first encountered and
+    automatically write 0 to the next entry and skip the next spin of the
+    loop. Thus, a separate counter or array appends to keep track of block
+    sizes are avoided. If needed, zeros can be filtered out later without
+    loosing the block structure.
+
+    Parameters
+    ----------
+    a : ndarray
+        The permutation array ipiv returned by LAPACK
+    lower : bool, optional
+        The switch to select whether upper or lower triangle is chosen in
+        the LAPACK call.
+
+    Returns
+    -------
+    swap_ : ndarray
+        The array that defines the row/column swap operations. For example,
+        if row two is swapped with row four, the result is [0, 3, 2, 3].
+    pivots : ndarray
+        The array that defines the block diagonal structure as given above.
+
+    """
+    n = a.size
+    swap_ = arange(n)
+    pivots = zeros_like(swap_, dtype=int)
+    skip_2x2 = False
+
+    # Some upper/lower dependent offset values
+    # range (s)tart, r(e)nd, r(i)ncrement
+    x, y, rs, re, ri = (1, 0, 0, n, 1) if lower else (-1, -1, n-1, -1, -1)
+
+    for ind in range(rs, re, ri):
+        # If previous spin belonged already to a 2x2 block
+        if skip_2x2:
+            skip_2x2 = False
+            continue
+
+        cur_val = a[ind]
+        # do we have a 1x1 block or not?
+        if cur_val > 0:
+            if cur_val != ind+1:
+                # Index value != array value --> permutation required
+                swap_[ind] = swap_[cur_val-1]
+            pivots[ind] = 1
+        # Not.
+        elif cur_val < 0 and cur_val == a[ind+x]:
+            # first neg entry of 2x2 block identifier
+            if -cur_val != ind+2:
+                # Index value != array value --> permutation required
+                swap_[ind+x] = swap_[-cur_val-1]
+            pivots[ind+y] = 2
+            skip_2x2 = True
+        else:  # Doesn't make sense, give up
+            raise ValueError('While parsing the permutation array '
+                             'in "scipy.linalg.ldl", invalid entries '
+                             'found. The array syntax is invalid.')
+    return swap_, pivots
+
+
+def _ldl_get_d_and_l(ldu, pivs, lower=True, hermitian=True):
+    """
+    Helper function to extract the diagonal and triangular matrices for
+    LDL.T factorization.
+
+    Parameters
+    ----------
+    ldu : ndarray
+        The compact output returned by the LAPACK routing
+    pivs : ndarray
+        The sanitized array of {0, 1, 2} denoting the sizes of the pivots. For
+        every 2 there is a succeeding 0.
+    lower : bool, optional
+        If set to False, upper triangular part is considered.
+    hermitian : bool, optional
+        If set to False a symmetric complex array is assumed.
+
+    Returns
+    -------
+    d : ndarray
+        The block diagonal matrix.
+    lu : ndarray
+        The upper/lower triangular matrix
+    """
+    is_c = iscomplexobj(ldu)
+    d = diag(diag(ldu))
+    n = d.shape[0]
+    blk_i = 0  # block index
+
+    # row/column offsets for selecting sub-, super-diagonal
+    x, y = (1, 0) if lower else (0, 1)
+
+    lu = tril(ldu, -1) if lower else triu(ldu, 1)
+    diag_inds = arange(n)
+    lu[diag_inds, diag_inds] = 1
+
+    for blk in pivs[pivs != 0]:
+        # increment the block index and check for 2s
+        # if 2 then copy the off diagonals depending on uplo
+        inc = blk_i + blk
+
+        if blk == 2:
+            d[blk_i+x, blk_i+y] = ldu[blk_i+x, blk_i+y]
+            # If Hermitian matrix is factorized, the cross-offdiagonal element
+            # should be conjugated.
+            if is_c and hermitian:
+                d[blk_i+y, blk_i+x] = ldu[blk_i+x, blk_i+y].conj()
+            else:
+                d[blk_i+y, blk_i+x] = ldu[blk_i+x, blk_i+y]
+
+            lu[blk_i+x, blk_i+y] = 0.
+        blk_i = inc
+
+    return d, lu
+
+
+def _ldl_construct_tri_factor(lu, swap_vec, pivs, lower=True):
+    """
+    Helper function to construct explicit outer factors of LDL factorization.
+
+    If lower is True the permuted factors are multiplied as L(1)*L(2)*...*L(k).
+    Otherwise, the permuted factors are multiplied as L(k)*...*L(2)*L(1). See
+    LAPACK documentation for more details.
+
+    Parameters
+    ----------
+    lu : ndarray
+        The triangular array that is extracted from LAPACK routine call with
+        ones on the diagonals.
+    swap_vec : ndarray
+        The array that defines the row swapping indices. If k'th entry is m
+        then rows k,m are swapped. Notice that m'th entry is not necessarily
+        k to avoid undoing the swapping.
+    pivs : ndarray
+        The array that defines the block diagonal structure returned by
+        _ldl_sanitize_ipiv().
+    lower : bool, optional
+        The boolean to switch between lower and upper triangular structure.
+
+    Returns
+    -------
+    lu : ndarray
+        The square outer factor which satisfies the L * D * L.T = A
+    perm : ndarray
+        The permutation vector that brings the lu to the triangular form
+
+    Notes
+    -----
+    Note that the original argument "lu" is overwritten.
+
+    """
+    n = lu.shape[0]
+    perm = arange(n)
+    # Setup the reading order of the permutation matrix for upper/lower
+    rs, re, ri = (n-1, -1, -1) if lower else (0, n, 1)
+
+    for ind in range(rs, re, ri):
+        s_ind = swap_vec[ind]
+        if s_ind != ind:
+            # Column start and end positions
+            col_s = ind if lower else 0
+            col_e = n if lower else ind+1
+
+            # If we stumble upon a 2x2 block include both cols in the perm.
+            if pivs[ind] == (0 if lower else 2):
+                col_s += -1 if lower else 0
+                col_e += 0 if lower else 1
+            lu[[s_ind, ind], col_s:col_e] = lu[[ind, s_ind], col_s:col_e]
+            perm[[s_ind, ind]] = perm[[ind, s_ind]]
+
+    return lu, argsort(perm)


Test patch:
diff --git a/scipy/linalg/tests/test_decomp_ldl.py b/scipy/linalg/tests/test_decomp_ldl.py
new file mode 100644
--- /dev/null
+++ b/scipy/linalg/tests/test_decomp_ldl.py
@@ -0,0 +1,137 @@
+from __future__ import division, print_function, absolute_import
+
+import itertools
+from numpy.testing import assert_array_almost_equal, assert_allclose, assert_
+from numpy import (array, eye, zeros, empty_like, empty, tril_indices_from,
+                   tril, triu_indices_from, spacing, float32, float64,
+                   complex64, complex128)
+from numpy.random import rand, randint, seed
+from scipy.linalg import ldl
+from pytest import raises as assert_raises, warns
+from numpy import ComplexWarning
+
+
+def test_args():
+    A = eye(3)
+    # Nonsquare array
+    assert_raises(ValueError, ldl, A[:, :2])
+    # Complex matrix with imaginary diagonal entries with "hermitian=True"
+    with warns(ComplexWarning):
+        ldl(A*1j)
+
+
+def test_empty_array():
+    a = empty((0, 0), dtype=complex)
+    l, d, p = ldl(empty((0, 0)))
+    assert_array_almost_equal(l, empty_like(a))
+    assert_array_almost_equal(d, empty_like(a))
+    assert_array_almost_equal(p, array([], dtype=int))
+
+
+def test_simple():
+    a = array([[-0.39-0.71j, 5.14-0.64j, -7.86-2.96j, 3.80+0.92j],
+               [5.14-0.64j, 8.86+1.81j, -3.52+0.58j, 5.32-1.59j],
+               [-7.86-2.96j, -3.52+0.58j, -2.83-0.03j, -1.54-2.86j],
+               [3.80+0.92j, 5.32-1.59j, -1.54-2.86j, -0.56+0.12j]])
+    b = array([[5., 10, 1, 18],
+               [10., 2, 11, 1],
+               [1., 11, 19, 9],
+               [18., 1, 9, 0]])
+    c = array([[52., 97, 112, 107, 50],
+               [97., 114, 89, 98, 13],
+               [112., 89, 64, 33, 6],
+               [107., 98, 33, 60, 73],
+               [50., 13, 6, 73, 77]])
+
+    d = array([[2., 2, -4, 0, 4],
+               [2., -2, -2, 10, -8],
+               [-4., -2, 6, -8, -4],
+               [0., 10, -8, 6, -6],
+               [4., -8, -4, -6, 10]])
+    e = array([[-1.36+0.00j, 0+0j, 0+0j, 0+0j],
+               [1.58-0.90j, -8.87+0j, 0+0j, 0+0j],
+               [2.21+0.21j, -1.84+0.03j, -4.63+0j, 0+0j],
+               [3.91-1.50j, -1.78-1.18j, 0.11-0.11j, -1.84+0.00j]])
+    for x in (b, c, d):
+        l, d, p = ldl(x)
+        assert_allclose(l.dot(d).dot(l.T), x, atol=spacing(1000.), rtol=0)
+
+        u, d, p = ldl(x, lower=False)
+        assert_allclose(u.dot(d).dot(u.T), x, atol=spacing(1000.), rtol=0)
+
+    l, d, p = ldl(a, hermitian=False)
+    assert_allclose(l.dot(d).dot(l.T), a, atol=spacing(1000.), rtol=0)
+
+    u, d, p = ldl(a, lower=False, hermitian=False)
+    assert_allclose(u.dot(d).dot(u.T), a, atol=spacing(1000.), rtol=0)
+
+    # Use upper part for the computation and use the lower part for comparison
+    l, d, p = ldl(e.conj().T, lower=0)
+    assert_allclose(tril(l.dot(d).dot(l.conj().T)-e), zeros((4, 4)),
+                    atol=spacing(1000.), rtol=0)
+
+
+def test_permutations():
+    seed(1234)
+    for _ in range(10):
+        n = randint(1, 100)
+        # Random real/complex array
+        x = rand(n, n) if randint(2) else rand(n, n) + rand(n, n)*1j
+        x = x + x.conj().T
+        x += eye(n)*randint(5, 1e6)
+        l_ind = tril_indices_from(x, k=-1)
+        u_ind = triu_indices_from(x, k=1)
+
+        # Test whether permutations lead to a triangular array
+        u, d, p = ldl(x, lower=0)
+        # lower part should be zero
+        assert_(not any(u[p, :][l_ind]), 'Spin {} failed'.format(_))
+
+        l, d, p = ldl(x, lower=1)
+        # upper part should be zero
+        assert_(not any(l[p, :][u_ind]), 'Spin {} failed'.format(_))
+
+
+def test_ldl_type_size_combinations():
+    seed(1234)
+    sizes = [100, 1000]
+    real_dtypes = [float32, float64]
+    complex_dtypes = [complex64, complex128]
+
+    for n, dtype in itertools.product(sizes, real_dtypes):
+        msg = ("Failed for size: {}, dtype: {}".format(n, dtype))
+
+        x = rand(n, n).astype(dtype)
+        x = x + x.T
+        x += eye(n, dtype=dtype)*dtype(randint(5, 1e6))
+
+        l, d1, p = ldl(x)
+        u, d2, p = ldl(x, lower=0)
+        rtol = 1e-4 if dtype is float32 else 1e-10
+        assert_allclose(l.dot(d1).dot(l.T), x, rtol=rtol, err_msg=msg)
+        assert_allclose(u.dot(d2).dot(u.T), x, rtol=rtol, err_msg=msg)
+
+    for n, dtype in itertools.product(sizes, complex_dtypes):
+        msg1 = ("Her failed for size: {}, dtype: {}".format(n, dtype))
+        msg2 = ("Sym failed for size: {}, dtype: {}".format(n, dtype))
+
+        # Complex hermitian upper/lower
+        x = (rand(n, n)+1j*rand(n, n)).astype(dtype)
+        x = x+x.conj().T
+        x += eye(n, dtype=dtype)*dtype(randint(5, 1e6))
+
+        l, d1, p = ldl(x)
+        u, d2, p = ldl(x, lower=0)
+        rtol = 1e-4 if dtype is complex64 else 1e-10
+        assert_allclose(l.dot(d1).dot(l.conj().T), x, rtol=rtol, err_msg=msg1)
+        assert_allclose(u.dot(d2).dot(u.conj().T), x, rtol=rtol, err_msg=msg1)
+
+        # Complex symmetric upper/lower
+        x = (rand(n, n)+1j*rand(n, n)).astype(dtype)
+        x = x+x.T
+        x += eye(n, dtype=dtype)*dtype(randint(5, 1e6))
+
+        l, d1, p = ldl(x, hermitian=0)
+        u, d2, p = ldl(x, lower=0, hermitian=0)
+        assert_allclose(l.dot(d1).dot(l.T), x, rtol=rtol, err_msg=msg2)
+        assert_allclose(u.dot(d2).dot(u.T), x, rtol=rtol, err_msg=msg2)