Pytorch: a similar process like reverse pooling and replicate padding?

AerysS: <p>I have a tensor <code>A</code> that has shape <code>(batch_size, width, height)</code>. Assume that it has these values:</p>
<pre><code class="lang-python">A = torch.tensor([[
                  [0, 1],
                  [1, 0]
]])
</code></pre>
<p>I am also given a number <code>K</code> that is a positive integer. Let <code>K=2</code> in this case. I want to do a process that is similar to reverse pooling and replicate padding. This is the expected output:</p>
<pre><code class="lang-python">B = torch.tensor([[
                [0, 0, 1, 1],
                [0, 0, 1, 1],
                [1, 1, 0, 0],
                [1, 1, 0, 0]             
]])
</code></pre>
<p>Explanation: for each element in <code>A</code>, we expand it to the matrix of shape <code>(K, K)</code>, and put it in the result tensor. We continue to do this with other elements, and let the stride between them equals to the kernel size (that is, <code>K</code>).</p>
<p>How can I do this in PyTorch? Currently, <code>A</code> is a binary mask, but it could be better if I can expand it to non-binary case.</p>

ptrblck: <p>I’m not sure if it would perfectly fit your use case, as I didn’t fully understand the desired behavior regarding the stride and kernel size, but the replication padding would match your output:</p>
<pre><code class="lang-python">A = torch.tensor([[
                  [0, 1],
                  [1, 0]
]]).float()

B = F.pad(A.unsqueeze(1), (1, 1, 1, 1), mode='replicate')
print(B)
&gt; tensor([[[[0., 0., 1., 1.],
            [0., 0., 1., 1.],
            [1., 1., 0., 0.],
            [1., 1., 0., 0.]]]])
</code></pre>
<p>You could <code>squeeze(1)</code> the additionally added dimension afterwards, if needed.</p>

AerysS: <p>Hi, after several testing I think <code>F.interpolate(a.float(), scale_factor=k, mode="nearest")</code> suits best. But when I use <code>timeit</code> it reports the worst run is ~90x slower than the best run, and sometimes I get only 8x or 10x. Is this a problem?</p>

ptrblck: <p>Is the 90x slowdown reported by <code>timeit</code> for the same method?<br>
If so, I would assume it should remove outliers. In doubt, you could manually profile it by adding proper warmup iterations and measure the time as an average for multiple iterations.</p>

AerysS: <p>Yes, it is reported by <code>timeit</code> for the same piece of code. Tensor is initialized and transformed through that single line.</p>

AerysS: <p>As a side note, I am still having this issue, and I think this inconsistency makes inconsistent in the code runtime, which is very bad in benchmarking.</p>
<p>To reproduce the result:</p>
<pre><code class="lang-python">import torch
import torch.nn.functional as F

%%timeit -n 1000000  # Jupyter
F.interpolate(torch.rand(100, 1, 16, 16).to("cuda").float(), scale_factor=2, mode="nearest")
</code></pre>

ptrblck: <p>This is expected, since the first iteration would initialize the CUDA context etc. and is thus slow.<br>
Also, to properly profile CUDA operations, you would have to synchronize the code manually, since CUDA ops are executed asynchronously, which is why we recommend to use <code>torch.utils.benchmark</code> to profile workloads (which adds warmup iterations, synchronizes, and averages the runtimes).</p>