Calculation and optimization of multivariate norm for RGB image

username: Serge_Sinitsa
cooked: <p>Hi,</p>
<p>Given an RGB image (shape of 1,3,H,W) I want to write an optimization procedure to find parameters of pixels’ multivariate norm (i.e., mu (shape of 1,3) and sigma (shape of 3,3)). In addition I wan’t to see the probabilities/density values as an image (shape of 1,1,H,W)</p>
<p>For an err (Image - Mean) and sigma (3x3 matrix) I preform the following:</p>
<pre><code class="lang-auto">def get_probs(err, sigma):
        ch, h, w = err.size()[1:]
        err_flat = torch.reshape(err, (h * w, ch))

        inv_sigma = torch.linalg.inv(sigma)

        c1 = 1. / (torch.sqrt((2 * torch.pi) ** (1. / ch) * torch.linalg.det(sigma) + 1e-8))

        mul = torch.matmul(err_flat, inv_sigma)
        mul = torch.einsum('ij,ij-&gt;i', mul, err_flat)

        res = c1 * (torch.exp(-0.5 * mul)).view(1, 1, h, w)

        return res
</code></pre>
<p>sigma and err are optimized outside of this code, with the loss below:</p>
<pre><code class="lang-auto">loss = (w * err).mean()
</code></pre>
<p>w is some function of res from above.</p>
<p>The output of this function is a weird duplicates of an image (see below).<br>
But if i’m running this function outside of optimization loop I get a what I expect -density values match to input “error image”.</p>
<p>I also tried to use torch probabilies:</p>
<pre><code class="lang-auto">res = torch.exp(MultivariateNormal(torch.zeros_like(err_flat), sigma.repeat(h*w, 1, 1)).log_prob(err_flat))
</code></pre>
<p>But result is the same…</p>
<p>What do I do wrong here?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cb73e937d1768057c3fec9061a4436e010b1001.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/6cb73e937d1768057c3fec9061a4436e010b1001" title="Screenshot 2023-09-21 104951"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6cb73e937d1768057c3fec9061a4436e010b1001_2_477x500.jpeg" alt="Screenshot 2023-09-21 104951" data-base62-sha1="fvKaiTBxc3MRcIEI5ulXjLWOJEJ" width="477" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6cb73e937d1768057c3fec9061a4436e010b1001_2_477x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6cb73e937d1768057c3fec9061a4436e010b1001_2_715x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cb73e937d1768057c3fec9061a4436e010b1001.jpeg 2x" data-dominant-color="C0CED4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-09-21 104951</span><span class="informations">741×776 72.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>You are interleaving the data in your reshape operation and would need to <code>permute</code> the axes before flattening them.</p>

username: Serge_Sinitsa
cooked: <p>What is the correct permute order?<br>
Is it only before flatten or also before final reshape?</p>

username: ptrblck
cooked: <p>You would always need to <code>permute</code> the tensor if you want to move dimensions, since <code>reshape</code> or <code>view</code> on non-neighboring dimensions will interleave the tensor as seen here:</p>
<pre><code class="lang-python">n, c, h, w = 1, 2, 3, 3

err = torch.arange(n*c*h*w).view(n, c, h, w)
print(err)
# tensor([[[[ 0,  1,  2],
#           [ 3,  4,  5],
#           [ 6,  7,  8]],

#          [[ 9, 10, 11],
#           [12, 13, 14],
#           [15, 16, 17]]]])

# wrong since it's interleaving the data
err_flat = torch.reshape(err, (n, h*w, c))
print(err_flat)
# tensor([[[ 0,  1],
#          [ 2,  3],
#          [ 4,  5],
#          [ 6,  7],
#          [ 8,  9],
#          [10, 11],
#          [12, 13],
#          [14, 15],
#          [16, 17]]])

# right as the channel dimension is permuted first
y = err.permute(0, 2, 3, 1)
print(y.shape)
# torch.Size([1, 3, 3, 2])

y = y.view(n, h*w, c)
print(y)
# tensor([[[ 0,  9],
#          [ 1, 10],
#          [ 2, 11],
#          [ 3, 12],
#          [ 4, 13],
#          [ 5, 14],
#          [ 6, 15],
#          [ 7, 16],
#          [ 8, 17]]])
</code></pre>

username: Serge_Sinitsa
cooked: <p>Thanks!</p>
<p>The proper flatten procedure is:</p>
<pre><code class="lang-auto">bs, ch, h, w = err.size()
err_flat = err.permute(0, 2, 3, 1).contiguous().view(bs, h * w, ch)
</code></pre>