Split network, clone and autograd

qwerty1024: <p>I’ve followed the few posts here about splitting networks and autograd, but if I plot my weights/parameters it suggests split layers are not learning properly.</p>
<p>My setup:</p>
<pre><code class="lang-auto">input = Variable(torch.FloatTensor(data_batch))
target = Variable(torch.FloatTensor(target_batch))

x = input[:,:-1]	 # last column of input needed at end
y = input[:, -1].reshape(batch, -1)

z = x.clone()

z = f(z) # common layer

z1 = z.clone() # branch 1
z1 = f(z1)

z2 = z.clone() # branch 2
z2 = f(z2)

output = z1 + y*z2 # output
</code></pre>
<p>Gradients (red) and weights for ten epochs, shared layer:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c8db2ee0ea66f0bf8408ffcd067aba6297124490.png" data-download-href="https://discuss.pytorch.org/uploads/default/c8db2ee0ea66f0bf8408ffcd067aba6297124490" title="grads-layer0-s"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8db2ee0ea66f0bf8408ffcd067aba6297124490_2_690x265.png" alt="grads-layer0-s" data-base62-sha1="sER5GImUx8cgjjBM3DRXMdUoQBG" width="690" height="265" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8db2ee0ea66f0bf8408ffcd067aba6297124490_2_690x265.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8db2ee0ea66f0bf8408ffcd067aba6297124490_2_1035x397.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/8/c8db2ee0ea66f0bf8408ffcd067aba6297124490.png 2x" data-dominant-color="FBF5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grads-layer0-s</span><span class="informations">1300×500 19.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/2/724defffa2beafaadf8c9cc159aca759328b2e45.png" data-download-href="https://discuss.pytorch.org/uploads/default/724defffa2beafaadf8c9cc159aca759328b2e45" title="w-layer0-s"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/2/724defffa2beafaadf8c9cc159aca759328b2e45.png" alt="w-layer0-s" data-base62-sha1="gjbpxvx7nmtdqTkVHnUYe8A2MbH" width="690" height="265" data-dominant-color="E0E0FB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">w-layer0-s</span><span class="informations">1300×500 8.04 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>For branch 1:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/7/37368d96c26d967a8bab430af8c85eb8e0c179cf.png" data-download-href="https://discuss.pytorch.org/uploads/default/37368d96c26d967a8bab430af8c85eb8e0c179cf" title="w-layer4-c"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/7/37368d96c26d967a8bab430af8c85eb8e0c179cf.png" alt="w-layer4-c" data-base62-sha1="7Sr8HmxBmU8QUuH9twJTM9M1Ibt" width="690" height="265" data-dominant-color="BCBCFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">w-layer4-c</span><span class="informations">1300×500 12.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/a/cafaddd1758de814288a4a84030cb15706b653ad.png" data-download-href="https://discuss.pytorch.org/uploads/default/cafaddd1758de814288a4a84030cb15706b653ad" title="grads-layer4-c"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/cafaddd1758de814288a4a84030cb15706b653ad_2_690x265.png" alt="grads-layer4-c" data-base62-sha1="sXDVx9weXzK0VxHYKA9IHV6PQip" width="690" height="265" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/cafaddd1758de814288a4a84030cb15706b653ad_2_690x265.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/cafaddd1758de814288a4a84030cb15706b653ad_2_1035x397.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/a/cafaddd1758de814288a4a84030cb15706b653ad.png 2x" data-dominant-color="FBD3D3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grads-layer4-c</span><span class="informations">1300×500 14.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>For branch 2:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b6e758ecbdce9832f06d5478ab8827c2478b52e.png" data-download-href="https://discuss.pytorch.org/uploads/default/7b6e758ecbdce9832f06d5478ab8827c2478b52e" title="grads-layer8-t_cloned"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b6e758ecbdce9832f06d5478ab8827c2478b52e_2_690x265.png" alt="grads-layer8-t_cloned" data-base62-sha1="hBVnYd7jj4Onwn2VY18AXZ1YWRw" width="690" height="265" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b6e758ecbdce9832f06d5478ab8827c2478b52e_2_690x265.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b6e758ecbdce9832f06d5478ab8827c2478b52e_2_1035x397.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b6e758ecbdce9832f06d5478ab8827c2478b52e.png 2x" data-dominant-color="FBCFCF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grads-layer8-t_cloned</span><span class="informations">1300×500 14.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/e/beea4fe1165c5200992ad25b833f19b29e2011c3.png" data-download-href="https://discuss.pytorch.org/uploads/default/beea4fe1165c5200992ad25b833f19b29e2011c3" title="w-layer8-t_cloned"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/e/beea4fe1165c5200992ad25b833f19b29e2011c3.png" alt="w-layer8-t_cloned" data-base62-sha1="reUJioe16IEoWW20zUT3fUpzGL1" width="690" height="265" data-dominant-color="BCBCFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">w-layer8-t_cloned</span><span class="informations">1300×500 7.35 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I suspect I’m not splitting properly, looks to me like the split layers are simply not learning. I have validated that cloning creates a new variable object in memory, but I’m missing something here.</p>
<p>Any insight appreciated.</p>

albanD: <p>Hi,</p>
<p>You don’t actually need to clone here.  You can re-use z as the input for <code>f</code> multiple times with no issues.<br>
Your weights do seem to change in both branches. Do you have any issue with the loss value?</p>

qwerty1024: <p>Got it, thanks. You are right that the weights change a bit across branches. And yes loss decreases extremely slow and not by much.</p>