Upsample+Conv2d vs ConvTranspose2d

max_mz: <p>Hi everyone,</p>
<p>There have been topics about the difference between torch.nn.Upsample and torch.nn.ConvTranspose2d but I don’t see anyone speaking about the difference between :</p>
<p>1 : <em>torch.nn.Upsample + torch.nn.Conv2d</em><br>
2 : <em>torch.nn.ConvTranspose2d</em></p>
<p>I am asking the question because i saw U-Net implementation using 1 and GAN/Autoencoder implementation using 2 but i don’t really know why.</p>
<p>If you have any hint i would really appreciate it.</p>

KFrank: <p>Hi Max!</p>
<aside class="quote no-group" data-username="max_mz" data-post="1" data-topic="138081" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/max_mz/48/33766_2.png" class="avatar"> max_mz:</div>
<blockquote>
<p>1 : <em>torch.nn.Upsample + torch.nn.Conv2d</em><br>
2 : <em>torch.nn.ConvTranspose2d</em></p>
</blockquote>
</aside>
<p><code>Upsample</code> plus <code>Conv2d</code> and <code>ConvTranspose2d</code> would do similar<br>
things, but they differ distinctly in detail.</p>
<p>Use <code>Upsample</code> (without <code>Conv2d</code>) if you want cheaper upsampling,<br>
but without trainable parameters, and use <code>ConvTranspose2d</code> if you<br>
want the trainable parameters.</p>
<p>Although you could add <code>Conv2d</code> to <code>Upsample</code> to get trainable<br>
parameters and similar functionality to <code>ConvTranspose2d</code>, you<br>
wouldn’t get the full benefit.</p>
<p>Consider <code>ConvTranspose2d</code> with a 2x2 kernel.  It, in effect, duplicates<br>
each pixel to form a 2x2 block of identical pixels, and then applies<br>
each of the 4 <code>in_channels</code> x <code>out_channels</code> slices in its 2x2<br>
<code>height</code> x <code>width</code> spatial array as single-pixel “depthwise” convolutions<br>
to each of the 4 pixels in the 2x2 block.</p>
<p>If, instead, you use <code>Upsample</code> to explicitly create that 2x2 block<br>
of identical (or non-identical, but similar interpolated pixels), and<br>
then apply a <code>Conv2d</code> with a 2x2 kernel, its 4 <code>in_channels</code> x<br>
<code>out_channels</code> depthwise convolutions get summed together,<br>
because they are summed across the 4 identical pixels in the 2x2<br>
block (or blended together if <code>Upsample</code> used interpolation).</p>
<p>So you have the same number of parameters and cost of training<br>
as using <code>ConvTranspose2d</code>, but you’ve thrown away (summed<br>
together) three-fourths of that expressiveness and trainability.</p>
<p>Best.</p>
<p>K. Frank</p>

max_mz: <p>Hi K. Frank,</p>
<p>Thanks for the fast and clear response.</p>
<p>If i understand correctly the 2 approaches can do vaguely the same thing. But to get an intuition of what’s different :</p>
<ul>
<li>With <code>Upsample</code> and <code>Conv2d</code> you will mix together pixels along space dimension.</li>
<li>Whereas if you only use <code>ConvTranspose2d</code> with <code>(2,2)</code> kernel and stride <code>(2,2)</code> the pixels are going to stay separated along the spatial dimension. (it would not be the case for a kernel size &gt; <code>(2,2)</code> but still the pixels would not be mixed as much as with <code>Upsample</code> and <code>Conv2d</code>)</li>
</ul>
<p>Have a good day,<br>
Max</p>

pu239: <p>for people that stumble upon this: read the <a href="https://distill.pub/2016/deconv-checkerboard/" rel="noopener nofollow ugc">distill.pub article</a> on this, their blogs were the best</p>