What formula is `torch.distributions.MultivariateNormal(loc, cov).sample()` using?

username: Alio
cooked: <p>I thought taking a sample from the Multivariate Normal distribution involved the equation below.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/6/56051a8a3c52368a96030c8c86a8224eac014e08.png" data-download-href="https://discuss.pytorch.org/uploads/default/56051a8a3c52368a96030c8c86a8224eac014e08" title="mvn"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/56051a8a3c52368a96030c8c86a8224eac014e08_2_690x116.png" alt="mvn" data-base62-sha1="cgXZOyYXNnTDP2H7Egx35qCEq48" width="690" height="116" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/56051a8a3c52368a96030c8c86a8224eac014e08_2_690x116.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/56051a8a3c52368a96030c8c86a8224eac014e08_2_1035x174.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/6/56051a8a3c52368a96030c8c86a8224eac014e08.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">mvn</span><span class="informations">1274×216 19.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But after digging into the <code>torch.distributions.MultivariateNormal(loc, cov).sample()</code> source code it looks like it is essentially just taking the Cholesky Decomposition of the covariance, doing a <code>torch.matmul</code> of the Cholesky by a random 1D array the length of the Cholesky and adding the <code>loc</code>.</p>
<p>Since my Statistics isn’t what it should be can you please explain why this is a Multivariate Normal Distribution? What is the equation for this? What is my equation? They definitely are not the same thing.</p>

username: KFrank
cooked: <p>Hi Alio!</p>
<aside class="quote no-group" data-username="Alio" data-post="1" data-topic="159160" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alio/48/8142_2.png" class="avatar"> Alio:</div>
<blockquote>
<p>it looks like it is essentially just taking the Cholesky Decomposition of the covariance, doing a <code>torch.matmul</code> of the Cholesky by a random 1D array the length of the Cholesky and adding the <code>loc</code>.</p>
</blockquote>
</aside>
<p>The random 1d array is a vector of independently sampled normal variates<br>
(with mean zero and standard deviation one).   That is the same as saying<br>
that it is a single sample from a multidimensional (with the number of<br>
dimensions equal to the length of the array) spherically-symmetrical<br>
multivariate normal distribution centered at the origin (multidimensional<br>
mean of zero) with its covariance matrix equal to the identity matrix (that<br>
is, standard deviations of one and no cross-correlations).</p>
<p>So we start with a multivariate normal distribution, but one with a special<br>
form.</p>
<p>The Cholesky decomposition of the input covariance matrix is, morally<br>
speaking, its “square root.”  Just as you would multiply a single normal<br>
deviate (with standard deviation one) with the square-root of a desired<br>
variance to produce a gaussian deviate with that desired variance, we<br>
multiply the sample from the multivariate normal distribution with covariance<br>
equal to the identity matrix with the square root of the desired covariance<br>
matrix.</p>
<p>This resulting multidimensional sample still has a (multidimensional) mean<br>
of zero.  Just as we would do in the one-dimensional case, we add to that<br>
sample the desired (multidimensional) mean to get the final sample.</p>
<p>This is the standard algorithm for generating samples from a multivariate<br>
gaussian distribution.  Note, you only have to perform the Cholesky<br>
decomposition once (and then reuse it), even if you generate a lot of<br>
samples.</p>
<p>Best.</p>
<p>K. Frank</p>