Covariance and gradient support

username: dem123456789
cooked: <p>Is there a proposal for torch.cov() and its gradient support?</p>

username: ModarTensai
cooked: <p>The covariance of <code>X</code> of shape <code>[*, n]</code> can be easily <a href="https://en.wikipedia.org/wiki/Covariance_matrix#Estimation" rel="nofollow noopener">estimated as</a>:<br>
<code>cov(X) = MM^T/(n-1)</code> where <code>M = X - columns_means(X)</code>.<br>
If we looked into the implementation of <a href="https://github.com/numpy/numpy/blob/master/numpy/lib/function_base.py#L2122-L2328" rel="nofollow noopener"><code>np.cov</code></a>, we can optimize it as follows:</p>
<pre><code class="lang-python">import warnings
import numpy as np

def np_cov(m, rowvar=False):
    # Handles complex arrays too
    m = np.asarray(m)
    if m.ndim &gt; 2:
        raise ValueError('m has more than 2 dimensions')
    dtype = np.result_type(m, np.float64)
    m = np.array(m, ndmin=2, dtype=dtype)
    if not rowvar and m.shape[0] != 1:
        m = m.T
    if m.shape[0] == 0:
        return np.array([], dtype=dtype).reshape(0, 0)

    # Determine the normalization
    fact = m.shape[1] - 1
    if fact &lt;= 0:
        warnings.warn("Degrees of freedom &lt;= 0 for slice",
                      RuntimeWarning, stacklevel=2)
        fact = 0.0

    m -= np.mean(m, axis=1, keepdims=1)
    c = np.dot(m, m.T.conj())
    c *= np.true_divide(1, fact)
    return c.squeeze()
</code></pre>
<p>This can be easily tested as follows:</p>
<pre><code class="lang-python">r = False
x = np.random.rand(3, 1000)
np_c = np.cov(x, rowvar=r)
our_c = np_cov(x, rowvar=r)
print(np.allclose(np_c, our_c))
</code></pre>
<p>To port it to pytorch, I did the following:</p>
<pre><code class="lang-python">import torch

def cov(m, rowvar=False):
    '''Estimate a covariance matrix given data.

    Covariance indicates the level to which two variables vary together.
    If we examine N-dimensional samples, `X = [x_1, x_2, ... x_N]^T`,
    then the covariance matrix element `C_{ij}` is the covariance of
    `x_i` and `x_j`. The element `C_{ii}` is the variance of `x_i`.

    Args:
        m: A 1-D or 2-D array containing multiple variables and observations.
            Each row of `m` represents a variable, and each column a single
            observation of all those variables.
        rowvar: If `rowvar` is True, then each row represents a
            variable, with observations in the columns. Otherwise, the
            relationship is transposed: each column represents a variable,
            while the rows contain observations.

    Returns:
        The covariance matrix of the variables.
    '''
    if m.dim() &gt; 2:
        raise ValueError('m has more than 2 dimensions')
    if m.dim() &lt; 2:
        m = m.view(1, -1)
    if not rowvar and m.size(0) != 1:
        m = m.t()
    # m = m.type(torch.double)  # uncomment this line if desired
    fact = 1.0 / (m.size(1) - 1)
    m -= torch.mean(m, dim=1, keepdim=True)
    mt = m.t()  # if complex: mt = m.t().conj()
    return fact * m.matmul(mt).squeeze()
</code></pre>
<p>It is maybe important here to note that currently pytorch <a href="https://github.com/pytorch/pytorch/issues/755" rel="nofollow noopener">doesn’t support complex tensors</a>, so the conjugate of a real matrix is itself. Another thing is that I like the pytorch convention where each function that consumes tensors and produces others outputs tensors of the same dtype. If you don’t desire this behaviour just uncomment the type casting line. Otherwise, cast your tensor to double before passing it to the function. Finally, I prefer <code>rowvar=False</code> by default which is not the default behaviour of <code>np.cov</code>.</p>
<p>Here is how to test this implementation:</p>
<pre><code class="lang-python">r = False
x = np.random.rand(3, 1000)
xt = torch.from_numpy(x).type(torch.double)
np_c = np.cov(x, rowvar=r)
our_c = cov(xt, rowvar=r).numpy()
print(np.allclose(np_c, our_c))
</code></pre>
<p>Because no special functions were used, we can acquire the gradient of this function with auto-differentiation.</p>

username: ModarTensai
cooked: <p>I came across <a href="https://stackoverflow.com/questions/51416825/calculate-covariance-matrix-for-complex-data-in-two-channels-no-complex-data-ty/51422002" rel="nofollow noopener">this on StackOverflow</a> which asks for computing the covariance matrix for complex tensors in torch.</p>

username: themightyoarfish
cooked: <p>Yeah the problem with that is that you can’t always work with the resulting matrices, i.e. I would have had to do fourier transform on the result, which of course doesn’t work since PyTorch doesn’t have complex yet.</p>

username: ModarTensai
cooked: <p>This is the same code with <code>inplace</code> flag.</p>
<pre><code class="lang-auto">import torch

def cov(m, rowvar=True, inplace=False):
    '''Estimate a covariance matrix given data.

    Covariance indicates the level to which two variables vary together.
    If we examine N-dimensional samples, `X = [x_1, x_2, ... x_N]^T`,
    then the covariance matrix element `C_{ij}` is the covariance of
    `x_i` and `x_j`. The element `C_{ii}` is the variance of `x_i`.

    Args:
        m: A 1-D or 2-D array containing multiple variables and observations.
            Each row of `m` represents a variable, and each column a single
            observation of all those variables.
        rowvar: If `rowvar` is True, then each row represents a
            variable, with observations in the columns. Otherwise, the
            relationship is transposed: each column represents a variable,
            while the rows contain observations.

    Returns:
        The covariance matrix of the variables.
    '''
    if m.dim() &gt; 2:
        raise ValueError('m has more than 2 dimensions')
    if m.dim() &lt; 2:
        m = m.view(1, -1)
    if not rowvar and m.size(0) != 1:
        m = m.t()
    # m = m.type(torch.double)  # uncomment this line if desired
    fact = 1.0 / (m.size(1) - 1)
    if inplace:
        m -= torch.mean(m, dim=1, keepdim=True)
    else:
        m = m - torch.mean(m, dim=1, keepdim=True)
    mt = m.t()  # if complex: mt = m.t().conj()
    return fact * m.matmul(mt).squeeze()
</code></pre>

username: neurosurg
cooked: <p>cor() and cov() for a 2D data matrix is easy to write.</p>
<p>However, TBH, it is not that straight-forward e.g. for a OHE-encoded tensor built from that 2D data matrix.</p>
<p>So, what about nice-written cor() and cov() for a tensor?</p>

username: ModarTensai
cooked: <p>This topic has gone stale. Please, check the latest updates on this <a href="https://github.com/pytorch/pytorch/issues/19037" rel="noopener nofollow ugc">issue</a>.</p>
<p>As a sneak peek, check this implementation that supports any tensor type:</p>
<pre><code class="lang-python">def cov(tensor, rowvar=True, bias=False):
    """Estimate a covariance matrix (np.cov)
    https://gist.github.com/ModarTensai/5ab449acba9df1a26c12060240773110
    """
    tensor = tensor if rowvar else tensor.transpose(-1, -2)
    tensor = tensor - tensor.mean(dim=-1, keepdim=True)
    factor = 1 / (tensor.shape[-1] - int(not bool(bias)))
    return factor * tensor @ tensor.transpose(-1, -2).conj()
</code></pre>