Random gapfilling with values from pixel distribution

username: vitusbenson
cooked: <p>I want to achieve the following and am wondering if there is a nice, efficient way in PyTorch:</p>
<p>Given an RGB Image as a 3D-Tensor with gaps at certain pixel locations, I want to fill these gaps with randomly drawn pixels from the distribution of existing pixels inside that image.</p>
<p>Does anyone have any idea? Thank you very much in advance!</p>
<p>Here is a small example, inside</p>
<pre><code class="lang-auto">a = torch.Tensor(
                  [[[3,4,0],[0,2,3],[1,3,0]],
                   [[1,3,0],[0,2,1],[1,2,0]],
                   [[3,3,0],[0,1,3],[1,2,0]]])
</code></pre>
<p>replace all triples <code>[0,0,0]</code> along the first axis (<code>axis = 0</code>) with randomly drawn triplets from the other triplets in <code>a</code> along the first axis.</p>
<p>So a possible solution would be</p>
<pre><code class="lang-auto">a_filled = torch.tensor(
                         [[[3,4,2],[1,2,3],[1,3,2]],
                          [[1,3,2],[1,2,1],[1,2,2]],
                          [[3,3,1],[1,1,3],[1,2,1]]])
</code></pre>
<p>another one would be</p>
<pre><code class="lang-auto">a_filled = torch.tensor(
                         [[[3,4,4],[3,2,3],[1,3,1]],
                          [[1,3,3],[2,2,1],[1,2,1]],
                          [[3,3,3],[2,1,3],[1,2,1]]])
</code></pre>

username: Kushaj
cooked: <p>What do you mean by the distribution of existing pixels?</p>

username: vitusbenson
cooked: <p>Here is a small example:</p>
<pre><code class="lang-auto">a = torch.Tensor(
                  [[[3,4,0],[0,2,3],[1,3,0]],
                   [[1,3,0],[0,2,1],[1,2,0]],
                   [[3,3,0],[0,1,3],[1,2,0]]])
</code></pre>
<p>I would like to replace all pixels <code>a[i,j,k] == [0,0,0]</code> with randomly drawn pixels<code> != [0,0,0]</code> from a (with replacement).<br>
It is important, that the three channels are not treated independently but jointly… (to keep the “colors”).<br>
Does this clear things up?</p>

username: Caruso
cooked: <p>How about this? But I’m not sure if you want the mean of all dims together or separately for the channels.</p>
<pre><code class="lang-auto">a = torch.tensor(
[[[3,4,0],[0,2,3],[1,3,0]],
[[1,3,0],[0,2,1],[1,2,0]],
[[3,3,0],[0,1,3],[1,2,0]]], dtype=torch.float)

a[a==0] = a.mean()
a = a.to(torch.long)
</code></pre>

username: vitusbenson
cooked: <p>Thank you for the suggestion!<br>
I don’t want to gapfill with the mean, rather I want to randomly sample the gap fillings from the distribution of the other pixels, so in this example i randomly want to draw pixels from the set  <code>{[3,1,3],[4,3,3],[2,2,1],[3,1,3],[1,1,1],[3,2,2]}</code> to fill the three gaps (<code>[0,0,0]</code>) in a.</p>
<p>So a possible solution would be</p>
<pre><code class="lang-auto">a_filled = torch.tensor(
                         [[[3,4,2],[1,2,3],[1,3,2]],
                          [[1,3,2],[1,2,1],[1,2,2]],
                          [[3,3,1],[1,1,3],[1,2,1]]])
</code></pre>
<p>another one would be</p>
<pre><code class="lang-auto">a_filled = torch.tensor(
                         [[[3,4,4],[3,2,3],[1,3,1]],
                          [[1,3,3],[2,2,1],[1,2,1]],
                          [[3,3,3],[2,1,3],[1,2,1]]])
</code></pre>

username: Caruso
cooked: <p>You can do this, but its really inefficient ^^</p>
<pre><code class="lang-auto">import random

a_set = torch.unique(a)
a[a==0] = torch.tensor([random.choice(a_set) for _ in range(len(a[a==0]))])
</code></pre>

username: vitusbenson
cooked: <p>I know, whatever comes out will likely be inefficient.<br>
Still your proposed solution is not what i desire. It just fills the three values inside each pixel with random values out of the set <code>{0,1,2,3,4}</code> assuming a uniform distribution. I don’t want that. I want to fill it with actual pixels fro my image <code>a</code>, which requires me to 1. keep the RGB-triplets together 2. not sample over unique triples but over all triplets.</p>

username: Caruso
cooked: <p>Oh i got i wrong…we are talking about the shape HWC, right? And those ‘faulty’ pixels triplets have the value <code>[0, 0, 0]</code>?</p>
<p>EDIT: Sry i mean CHW ^^</p>

username: vitusbenson
cooked: <p>actually i thought my example is CHW, but does not matter. Yes the faulty pixels have value <code>[0,0,0]</code>.</p>

username: Kushaj
cooked: <p>For every 0 triplet, sample from a gaussian with mean and std from neighbouring pixels. You can consider a 5x5 window pixels around that (0,0,0) pixel and get mean, std from them.</p>

username: vitusbenson
cooked: <p>hmm, is a possibility but also misses the covariances between the channels RGB. An example: Say my image has either purple-ish or green-ish pixels of different brightness, so pixels of type (0,g,0) or (p,0,p). Now if i follow your approach i could end up with a pixel (p,g,p), which is not in the desired distribution of the image.</p>

username: Kushaj
cooked: <aside class="quote no-group" data-username="vitusbenson" data-post="11" data-topic="82825">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/d9b06d/48.png" class="avatar"> vitusbenson:</div>
<blockquote>
<p>but also misses the covariances between the channels RGB</p>
</blockquote>
</aside>
<p>Does this matter much nowadays. CNN is pretty robust to these small differences.</p>