Sampling from non-identity covariance matrix, but not satisfying constraint PositiveDefinite()

Pericules: <p>I’m trying to use <code>torch.distributions.multivariate_normal</code> to sample from a non-identity covariance normal distribution. My covariance matrix is 4096x4096, and I have estimated it from data.</p>
<p>However, when I try and run <code>m = multivariate_normal.MultivariateNormal(torch.zeros(4096), cov)</code>, I get the following:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [11], in &lt;cell line: 1&gt;()
----&gt; 1 m = multivariate_normal.MultivariateNormal(torch.zeros(4096), cov)

File /ext3/miniconda3/lib/python3.9/site-packages/torch/distributions/multivariate_normal.py:177, in MultivariateNormal.__init__(self, loc, covariance_matrix, precision_matrix, scale_tril, validate_args)
    174 self.loc = loc.expand(batch_shape + (-1,))
    176 event_shape = self.loc.shape[-1:]
--&gt; 177 super().__init__(batch_shape, event_shape, validate_args=validate_args)
    179 if scale_tril is not None:
    180     self._unbroadcasted_scale_tril = scale_tril

File /ext3/miniconda3/lib/python3.9/site-packages/torch/distributions/distribution.py:68, in Distribution.__init__(self, batch_shape, event_shape, validate_args)
     66         valid = constraint.check(value)
     67         if not valid.all():
---&gt; 68             raise ValueError(
     69                 f"Expected parameter {param} "
     70                 f"({type(value).__name__} of shape {tuple(value.shape)}) "
     71                 f"of distribution {repr(self)} "
     72                 f"to satisfy the constraint {repr(constraint)}, "
     73                 f"but found invalid values:\n{value}"
     74             )
     75 super().__init__()

ValueError: Expected parameter covariance_matrix (Tensor of shape (4096, 4096)) of distribution MultivariateNormal(loc: torch.Size([4096]), covariance_matrix: torch.Size([4096, 4096])) to satisfy the constraint PositiveDefinite(), but found invalid values:
tensor([[1.0383, 0.9125, 0.6980,  ..., 0.4600, 0.6637, 0.8524],
        [0.9125, 1.0187, 0.8847,  ..., 0.2892, 0.4774, 0.6710],
        [0.6980, 0.8847, 0.9858,  ..., 0.1393, 0.2974, 0.4747],
        ...,
        [0.4600, 0.2892, 0.1393,  ..., 0.9690, 0.8648, 0.6774],
        [0.6637, 0.4774, 0.2974,  ..., 0.8648, 0.9972, 0.8932],
        [0.8524, 0.6710, 0.4747,  ..., 0.6774, 0.8932, 1.0227]])
</code></pre>
<p>The matrix is symmetric however - when I evaluate: <code>(cov==cov.T).sum()</code> I get <code>16777216</code> which is 4906**2. When I compute the eigenvalues however, using:</p>
<pre><code class="lang-auto">eigs=torch.linalg.eigvals(covmat)
min(eigs.real)
</code></pre>
<p>I get a negative value: <code>tensor(-5.7518e-06)</code>. Perhaps because the matrix is large, there is some numerical error in the eigenvalue computation which is leading to a negative number, and therefore failing the pytorch checks? I notice when I subsample the matrix even slightly, to 4000x4000 instead of 4096, I am able to initialise the torch <code>MultivariateNormal</code> object without any errors.</p>
<p>Does anyone have a suggested workaround? Does it make sense for torch to use a numerically sensitive assertion here, when simply checking if the matrix is symmetric might also suffice?</p>

KFrank: <p>Hi Chris!</p>
<aside class="quote no-group" data-username="Pericules" data-post="1" data-topic="206543" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/b9e5f3/48.png" class="avatar"> Pericules:</div>
<blockquote>
<p>My covariance matrix is 4096x4096</p>
</blockquote>
</aside>
<p>This is pretty large, so you are likely asking for numerical issues.</p>
<aside class="quote no-group">
<blockquote>
<p>and I have estimated it from data.</p>
</blockquote>
</aside>
<p>Note that such an estimate will differ from the “true” population covariance.</p>
<p>I assume that you perform your estimate in a way that will give you a<br>
<em>mathematically</em> positive-definite covariance matrix.  Nonetheless, with<br>
a matrix that size there is a good chance that it will be numerically<br>
non-positive-definite.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">ValueError: Expected parameter covariance_matrix (Tensor of shape (4096, 4096)) of distribution MultivariateNormal(loc: torch.Size([4096]), covariance_matrix: torch.Size([4096, 4096])) to satisfy the constraint PositiveDefinite()
</code></pre>
<p>…</p>
<pre><code class="lang-auto">eigs=torch.linalg.eigvals(covmat)
min(eigs.real)
</code></pre>
<p>I get a negative value: <code>tensor(-5.7518e-06)</code>. Perhaps because the matrix is large, there is some numerical error in the eigenvalue computation which is leading to a negative number</p>
</blockquote>
</aside>
<p>This is possible.  I think it more likely, though, that your matrix – which is a<br>
floating-point approximation to your “true” matrix – does have a negative<br>
eigenvalue.  But, regardless, it’s more or less the same issue.</p>
<aside class="quote no-group">
<blockquote>
<p>I notice when I subsample the matrix even slightly, to 4000x4000 instead of 4096, I am able to initialise the torch <code>MultivariateNormal</code> object without any errors.</p>
</blockquote>
</aside>
<p><em>Slightly</em> reducing the size of the matrix seems fragile.  I could well believe<br>
that a negative eigenvalue would show up again were you to run a slightly<br>
different dataset.</p>
<aside class="quote no-group">
<blockquote>
<p>Does anyone have a suggested workaround?</p>
</blockquote>
</aside>
<p>You <em>might</em> be able to get away with performing this part of the computation<br>
in double precision.  But you might just end up again with a negative<br>
eigenvalue, but with a value of about <code>-1.e-14</code>.</p>
<p>Without knowing your use case, it’s hard to say.  What do you use the<br>
samples you draw from <code>MultivariateNormal</code> for?</p>
<p>You might be able to “regularize” your covariance matrix by adding a<br>
small “epsilon”, say, <code>1.e-5</code>, to its diagonal.  You could explicitly reduce<br>
the size of the matrix by keeping only the top 100 or 1000 or 2000<br>
eigenvectors that correspond to the largest eigenvalues (“principal<br>
components”).</p>
<p>You might also explore <a href="https://pytorch.org/docs/2.3/distributions.html#torch.distributions.lowrank_multivariate_normal.LowRankMultivariateNormal" rel="noopener nofollow ugc">LowRankMultivariateNormal.</a></p>
<p>Best.</p>
<p>K. Frank</p>