Spearman's Correlation

haltaha: <p>I need to use a rank-based correlation (Spearman’s Correlation) to compute my loss.<br>
Could anyone please help me with the implementation of Spearman’s Correlation between two vectors in Pytorch?</p>

haltaha: <p>In case someone is trying to look for a solution:</p>
<pre><code class="lang-auto">def cov(m):
    # m = m.type(torch.double)  # uncomment this line if desired
    fact = 1.0 / (m.shape[-1] - 1)  # 1 / N
    m -= torch.mean(m, dim=(1, 2), keepdim=True)
    mt = torch.transpose(m, 1, 2)  # if complex: mt = m.t().conj()
    return fact * m.matmul(mt).squeeze()


def compute_rank_correlation(x, y):
    x, y = rankmin(x), rankmin(y)
    return corrcoef(x, y)


def corrcoef(x, y):
    batch_size = x.shape[0]
    x = torch.stack((x, y), 1)
    # calculate covariance matrix of rows
    c = cov(x)
    # normalize covariance matrix
    d = torch.diagonal(c, dim1=1, dim2=2)
    stddev = torch.pow(d, 0.5)
    stddev = stddev.repeat(1, 2).view(batch_size, 2, 2)
    c = c.div(stddev)
    c = c.div(torch.transpose(stddev, 1, 2))
    return c[:, 1, 0]
</code></pre>
<p>There is small difference between the above implementation and scipy’s spearmanr implementation:</p>
<ol>
<li>This operate on batches, that is, it expects input to be of dimension (N, x) where N is the batch size for both parameters x and y.</li>
<li>scipy uses rank average while this uses rank min to compute the ranking before corrcoef. There is 10^-3 disagreement and although I suspect that this is the cause, this could possibly be.</li>
</ol>
<p>Please feel free to share your improvements on this as it could be useful for other people and me</p>

KFrank: <p>Hi haltaha!</p>
<aside class="quote no-group" data-username="haltaha" data-post="1" data-topic="91931" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/c68b51/48.png" class="avatar"> haltaha:</div>
<blockquote>
<p>I need to use a rank-based correlation (Spearman’s Correlation) to compute my loss.</p>
</blockquote>
</aside>
<p>Presumably you will use your “loss” to train your network with<br>
backpropagation by calling <code>loss.backward()</code>.</p>
<p>Because the ranks of your data values are discrete integers, the<br>
Spearman’s-rank-correlation loss you propose calculating will not<br>
be (usefully) differentiable, so you will not be able to backpropagate<br>
nor train.</p>
<aside class="quote no-group">
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">    x, y = rankmin(x), rankmin(y)
</code></pre>
</blockquote>
</aside>
<p>You don’t show us the code for <code>rankmin()</code>, but presumably buried<br>
in there somewhere is a non-differentiable call that returns the indices<br>
for the sorted values of <code>x</code> (sometimes referred to as <code>argsort()</code>).</p>
<p>Best.</p>
<p>K. Frank</p>

haltaha: <p>You are correct, thank you.</p>

zakajd: <p>Hi</p>
<p>Here is simplified version for computation of Spearman’s rank correlation coefficient. It’s non differentiable, but much faster than <code>scipy.stats.spearmanr</code> version.</p>
<pre><code class="lang-python">def _get_ranks(x: torch.Tensor) -&gt; torch.Tensor:
    tmp = x.argsort()
    ranks = torch.zeros_like(tmp)
    ranks[tmp] = torch.arange(len(x))
    return ranks

def spearman_correlation(x: torch.Tensor, y: torch.Tensor):
    """Compute correlation between 2 1-D vectors
    Args:
        x: Shape (N, )
        y: Shape (N, )
    """
    x_rank = _get_ranks(x)
    y_rank = _get_ranks(y)
    
    n = x.size(0)
    upper = 6 * torch.sum((x_rank - y_rank).pow(2))
    down = n * (n ** 2 - 1.0)
    return 1.0 - (upper / down)
</code></pre>
<p>Time comparison</p>
<pre><code class="lang-auto">x = torch.rand(1000)
y = torch.rand(1000)
%timeit spearman_correlation(x, y)
206 µs ± 15.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit spearmanr(x_n, y_n)[0]
592 µs ± 2.22 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)</code></pre>

avish: <p>Hi, I found a differential version at post:</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://forum.numer.ai/uploads/default/optimized/1X/bea71c23d60806887f6fecacf14c1c7fcf1df935_2_32x32.png" class="site-icon" width="16" height="16">

      <a href="https://forum.numer.ai/t/differentiable-spearman-in-pytorch-optimize-for-corr-directly/2287" target="_blank" rel="noopener nofollow ugc" title="04:21AM - 11 March 2021">Numerai Forum – 11 Mar 21</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:129/153;"><img src="https://forum.numer.ai/uploads/default/original/1X/bea71c23d60806887f6fecacf14c1c7fcf1df935.png" class="thumbnail" width="129" height="153"></div>

<h3><a href="https://forum.numer.ai/t/differentiable-spearman-in-pytorch-optimize-for-corr-directly/2287" target="_blank" rel="noopener nofollow ugc">Differentiable Spearman in PyTorch (Optimize for CORR directly)</a></h3>

  <p>@mdo previously showed how to use a custom loss function which involved taking the gradient of the sharpe ratio of the Pearson correlations over different eras. Although Pearson and Spearman might return similar values, it could be rewarding to...</p>

  <p>
    <span class="label1">Reading time: 4 mins 🕑</span>
      <span class="label2">Likes: 36 ❤</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Copying from above post:<br>
" A recent paper, <a href="https://github.com/google-research/fast-soft-sort/" rel="noopener nofollow ugc">Fast Differentiable Sorting and Ranking </a>, introduced a novel method for differentiable sorting and ranking, with the added bonus of O(nlogn) complexity (I would encourage reading the paper to learn more). We can leverage their open sourced code <a href="https://github.com/google-research/fast-soft-sort/" rel="noopener nofollow ugc">google-research/fast-soft-sort </a> in order to implement a differentiable version of the Spearman"</p>