Re-using layers in model?

ajhanwar: <p>Hi all, I am learning about PyTorch and have been practicing through implementing the YOLOv3 model.</p>
<p>The model itself has layers which are “repeated” in a sense. For example, we repeat the process of doing the following several times:</p>
<pre><code class="lang-auto">nn.Conv2d(256, 512, kernel_size=1, stride=1, padding=1)
nn.Conv2d(512, 256, kernel_size=3, stride=1, padding=1)
nn.Conv2d(256, 512, kernel_size=1, stride=1, padding=1)
nn.Conv2d(512, 256, kernel_size=3, stride=1, padding=1)
...
</code></pre>
<p>In such a situation would we need to explicitly list out each convolutional layer within the model definition? I understand each layer has an associated weight tensor and so I am not sure if we can simply re-use the layers.</p>

tymokvo: <p>Welcome to the forums!</p>
<p>Yes it is definitely possible! Whether or not you want to do it is a design decision. Broadly, it will lower memory cost while also reducing the degrees of freedom of the model.<br>
One of the important contributions of <a href="https://arxiv.org/abs/1608.06993" rel="noopener nofollow ugc">DenseNet</a> was the discovery that (for classification) you can pass the parameter tensors of early layers as inputs to deeper layers to reduce the memory cost of the model and encourage the model to learn features that are applicable at several scales.</p>
<p>You could share the parameters themselves like:</p>
<pre><code class="lang-auto">c = nn.Conv2d(3, 3, (3, 3))

im = torch.randn(1, 3, 8, 8, requires_grad=True)

o = c(im)
o = c(o)
</code></pre>
<p>And a render of its backprop graph looks like:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/d227844c23e50fc970779c8fc76c31d6903192ce.png" data-download-href="https://discuss.pytorch.org/uploads/default/d227844c23e50fc970779c8fc76c31d6903192ce" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d227844c23e50fc970779c8fc76c31d6903192ce_2_295x249.png" alt="image" data-base62-sha1="tZ6VQUwgNYfb3fK8HYLpUfmZFp4" width="295" height="249" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d227844c23e50fc970779c8fc76c31d6903192ce_2_295x249.png, https://discuss.pytorch.org/uploads/default/optimized/2X/d/d227844c23e50fc970779c8fc76c31d6903192ce_2_442x373.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/d/d227844c23e50fc970779c8fc76c31d6903192ce_2_590x498.png 2x" data-dominant-color="E1E7E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">985×833 44.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Or a DenseNet idea like:</p>
<pre><code class="lang-auto">c = nn.Conv2d(3, 3, (3, 3), 1, 1, bias=False)
u = nn.UpsamplingBilinear2d(scale_factor=3)

im = torch.randn(1, 3, 9, 9, requires_grad=True)
fms = u(c.weight)

o = c(im)
o = torch.cat([fms, o])
</code></pre>
<p>Which looks like:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/4dc405e705d2580a12a832cc169534df40d015f7.png" data-download-href="https://discuss.pytorch.org/uploads/default/4dc405e705d2580a12a832cc169534df40d015f7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4dc405e705d2580a12a832cc169534df40d015f7_2_345x203.png" alt="image" data-base62-sha1="b5WK2SR8L5amX9fxGUByGjMMJzF" width="345" height="203" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4dc405e705d2580a12a832cc169534df40d015f7_2_345x203.png, https://discuss.pytorch.org/uploads/default/optimized/2X/4/4dc405e705d2580a12a832cc169534df40d015f7_2_517x304.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/4/4dc405e705d2580a12a832cc169534df40d015f7_2_690x406.png 2x" data-dominant-color="EAEDEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1380×812 45.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am not sure if YOLO uses parameter sharing. I do seem to remember that YOLOv3 does use residual connections. But it would be an interesting experiment to see if the parameter sharing improves performance!</p>

ajhanwar: <p>Thank you so much for the detailed explanation!<br>
Just to confirm my understanding, reusing layers would effectively count as parameter sharing (using same weights)? And if so, is there a way to prevent such an occurrence without having to explicitly list out every layer?</p>

ptrblck: <p>Yes, reusing a layer would use the same parameters in its operation.<br>
If you are dealing with a lot of (simple to initialize) layers, you could try to create them in a loop and append each one to a <code>nn.ModuleList</code>.<br>
Later on you could pass the layers from this list to an <code>nn.Sequential</code> module or just call it again in a loop in your <code>forward</code> method.</p>