How to calculate the multivariate normal distribution using pytorch and math?

John_Price: <p>The mutivariate normal distribution is given as<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f7b9bdb804284a5e26b1fbfb435ebdf5817edcb.png" alt="multivariate_normal_distribution" data-base62-sha1="dCGbNL4fMw69Eo4tvxzd3cCHTh9" width="489" height="75"><br>
The formula can be calculated using numpy for example the following way:</p>
<pre><code class="lang-auto">def multivariate_normal_distribution(x, d, mean, covariance):
  x_m = x - mean
  return (1.0 / (np.sqrt((2 * np.pi)**d * np.linalg.det(covariance))) * np.exp(-(np.linalg.solve(covariance, x_m).T.dot(x_m)) / 2))
</code></pre>
<p>I want to do the same calculation but instead of using numpy I want to use pytorch and math. The idea is the following:</p>
<pre><code class="lang-auto">def multivariate_normal_distribution(x, d, mean, covariance):
  x_m = x - mean
  return (1.0 / (math.sqrt((2 * math.pi)**d * torch.linalg.det(covariance))) * torch.exp(-(np.linalg.solve(covariance, x_m).T.dot(x_m)) / 2))
</code></pre>
<p>The problem is in <code>np.linalg.solve()</code>. I know that with <code>.solve</code> it is possible to solve a linear matrix equation or a system of linear scalar equations. So I want to know if there is a pytorch function that does exactly the same? Help is much appriciated</p>

InnovArul: <p>Did you try this?<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/distributions.html#multivariatenormal" target="_blank" rel="noopener">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/distributions.html#multivariatenormal" target="_blank" rel="noopener">Probability distributions - torch.distributions — PyTorch 1.7.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<pre><code class="lang-auto">from  torch.distributions import multivariate_normal
dist = multivariate_normal.MultivariateNormal(loc=torch.zeros(5), covariance_matrix=torch.eye(5))
probability = torch.exp(dist.log_prob(torch.randn(5)))

</code></pre>

John_Price: <p>Thanks a lot, I will try that. But do you know if there is a function in pytorch which does the same like <code>np.linalg.solve</code> in the above formula? Thank you very much!</p>

KFrank: <p>Hi John!</p>
<aside class="quote no-group" data-username="John_Price" data-post="4" data-topic="105943" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/c67d28/48.png" class="avatar"> John_Price:</div>
<blockquote>
<p>But do you know if there is a function in pytorch which does the same like <code>np.linalg.solve</code> in the above formula?</p>
</blockquote>
</aside>
<p><a href="https://pytorch.org/docs/1.7.0/generated/torch.solve.html" rel="noopener nofollow ugc">torch.solve</a> is pytorch’s general linear-equation solver.</p>
<p>Your <code>covariance</code> should be symmetric positive definite, so, if you<br>
cared, you could go the Cholesky- factorization route and use:</p>
<p><a href="https://pytorch.org/docs/1.7.0/generated/torch.cholesky.html" rel="noopener nofollow ugc">torch.cholesky</a><br>
<a href="https://pytorch.org/docs/1.7.0/generated/torch.cholesky_solve.html" rel="noopener nofollow ugc">torch.cholesky_solve</a></p>
<p>(You could also pre-compute the <code>covariance</code>’s inverse with<br>
<a href="https://pytorch.org/docs/1.7.0/generated/torch.inverse.html" rel="noopener nofollow ugc">torch.inverse</a> or <a href="https://pytorch.org/docs/1.7.0/generated/torch.cholesky_inverse.html" rel="noopener nofollow ugc">torch.cholesky_inverse</a>.)</p>
<p>But unless you have an unusual use case, I would go with Arul’s<br>
suggestion and use<br>
<code>torch.distributions.multivariate_normal.MultivariateNormal</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

John_Price: <aside class="quote no-group" data-username="John_Price" data-post="1" data-topic="105943">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/c67d28/48.png" class="avatar"> John_Price:</div>
<blockquote>
<pre><code class="lang-auto">def multivariate_normal_distribution(x, d, mean, covariance):
  x_m = x - mean
  return (1.0 / (math.sqrt((2 * math.pi)**d * torch.linalg.det(covariance))) * torch.exp(-(np.linalg.solve(covariance, x_m).T.dot(x_m)) / 2))
</code></pre>
</blockquote>
</aside>
<p>Hi <a class="mention" href="/u/kfrank">@KFrank</a></p>
<p>so you mean, that it I have given a symmetric positive definite covariance, I could also write:</p>
<p>torch.exp(-(torch.cholesky_solve(covariance, x_m).T.dot(x_m)))</p>
<p>for the right term with the exponential?</p>
<p>Thank you very much for your help!</p>

John_Price: <p><a class="mention" href="/u/innovarul">@InnovArul</a>  I have got still one question. Is the part for the Mahalanobis-distance in the formula you wrote:<br>
<code>dist = multivariate_normal.MultivariateNormal(loc=torch.zeros(5), covariance_matrix=torch.eye(5))</code><br>
the same as<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/2/62265355544b752fdd03e8146845f68a253488c7.png" alt="m_distance" data-base62-sha1="e0gU3z6QteDYLAytMZGYkSg5zJZ" width="198" height="49"></p>
<p>because in literature the Mahalanobis-distance is given with square root instead of -0.5 as a factor</p>

KFrank: <p>Hi John!</p>
<aside class="quote no-group" data-username="John_Price" data-post="6" data-topic="105943" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/c67d28/48.png" class="avatar"> John_Price:</div>
<blockquote>
<p>so you mean, that it I have given a symmetric positive definite covariance, I could also write:</p>
<p>torch.exp(-(torch.cholesky_solve(covariance, x_m).T.dot(x_m)))</p>
</blockquote>
</aside>
<p>Basically, but <code>torch.cholesky_solve()</code> takes the Cholesky factor matrix,<br>
so you have to do it in two steps.</p>
<p>Here is an illustrative script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__

_ = torch.random.manual_seed (2020)

ma = torch.randn ((5, 5))
mspd = ma @ ma.T                           # make an SPD matrix
v = torch.randn ((5, 1))                   # make a vector
solveA = torch.solve (v, mspd)[0]          # solve direct
mcho = torch.cholesky (mspd)               # Cholesky factorization
solveB = torch.cholesky_solve (v, mcho)    # solve with Cholesky factorization
print ('solveA =', solveA)
print ('solveB =', solveB)
print ('solveA - solveB =', solveA - solveB)
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.6.0'
&gt;&gt;&gt; _ = torch.random.manual_seed (2020)
&gt;&gt;&gt; ma = torch.randn ((5, 5))
&gt;&gt;&gt; mspd = ma @ ma.T                           # make an SPD matrix
&gt;&gt;&gt; v = torch.randn ((5, 1))                   # make a vector
&gt;&gt;&gt; solveA = torch.solve (v, mspd)[0]          # solve direct
&gt;&gt;&gt; mcho = torch.cholesky (mspd)               # Cholesky factorization
&gt;&gt;&gt; solveB = torch.cholesky_solve (v, mcho)    # solve with Cholesky factorization
&gt;&gt;&gt; print ('solveA =', solveA)
solveA = tensor([[ 6.6744],
        [-1.5211],
        [ 1.1208],
        [ 1.1884],
        [-3.9746]])
&gt;&gt;&gt; print ('solveB =', solveB)
solveB = tensor([[ 6.6743],
        [-1.5211],
        [ 1.1208],
        [ 1.1884],
        [-3.9745]])
&gt;&gt;&gt; print ('solveA - solveB =', solveA - solveB)
solveA - solveB = tensor([[ 1.7643e-05],
        [-4.2915e-06],
        [ 2.1458e-06],
        [ 3.0994e-06],
        [-9.2983e-06]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

InnovArul: <p>You can take a look at the source code here if it helps:<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/_modules/torch/distributions/multivariate_normal.html#MultivariateNormal" target="_blank" rel="noopener">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/_modules/torch/distributions/multivariate_normal.html#MultivariateNormal" target="_blank" rel="noopener">torch.distributions.multivariate_normal — PyTorch 1.7.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>They calculate squared Mahalanobis distance in <code>_batch_mahalanobis()</code>.</p>

John_Price: <p><a class="mention" href="/u/kfrank">@KFrank</a> I should have read your last post a little closer because you wrote “Cholesky- factorization route” and added both functions. Thank you for the helpful code snippet</p>
<p>So in my case I just have to write:</p>
<pre><code class="lang-auto">tmp = torch.cholesky(covariance)
res = torch.cholesky_solve(x_m, tmp)
</code></pre>
<p>Does it matter if <code>cholesky_solve()</code> takes first the vector and then the matrix or the other way round? And what if x_m is a distance matrix where the entries are eucledian distances? I know that Mahalanobis-distances are there  to calculate the distances between a data point and a distribution<br>
but is it desired that we first calculate a Euclidean distance with x_m and then subsequently calculate the Mahalanobis distance?</p>

fbordignon: <p>Hey guys, thanks for the replies. In my case I have a single covariance matrix of a neighborhood (kernel) of a given pixel of an image and I need to compute the log_prob for all the pixels in the image using the neighborhood for them. i.e. for each pixel i,j I have to use a fixed and previously computed covariance matrix and mean to compute the multivariate normal log_prob that uses the neighbor pixels [i±k,j±k] being k the kernel size. One approach is to use torch.tensor.unfold to generate a view of the image tensor with the windows matching the kernel size, reshape and pass it to log_prob from the Multivariate normal. This approach works but it uses a lot of memory. Would anyone know a faster/better way to do that?</p>