Network with chained identical blocks

Albert65: <p>Dear All,</p>
<p>I am trying to build a network where identical blocks are chained such that the next block uses the output of the previous one. Let me give an example.</p>
<ol>
<li>Suppose, my block is a multi-layer perceptron (MLP) with 10 inputs and single output, trained to solve regression problem.</li>
<li>There are 3 blocks in total.</li>
<li>The input is represented by 3x10 data matrix M.</li>
<li>The first MLP takes the first row M[0,:] of the data matrix as an input and produces the output z1.</li>
<li>The second MLP takes the second row M[1,:] except one entry (say, the 1st one). Instead, z1 value is used in place of omitted entry M[1,0]. This second MLP outputs z2.</li>
<li>The third MLP takes the third row except two entries (say, the 1st and 2nd ones). Instead, z1 and z2 values are used in place of omitted entries M[2,0] and M[2,1]. This third MLP outputs z3.</li>
</ol>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d7d37b3d2189695e2315535c3e222e3f73039c2.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/0d7d37b3d2189695e2315535c3e222e3f73039c2" title="example_nn"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d7d37b3d2189695e2315535c3e222e3f73039c2_2_690x440.jpeg" alt="example_nn" data-base62-sha1="1Vku2Ed4twFKinNcqMQAQFPpsFs" width="690" height="440" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d7d37b3d2189695e2315535c3e222e3f73039c2_2_690x440.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d7d37b3d2189695e2315535c3e222e3f73039c2.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d7d37b3d2189695e2315535c3e222e3f73039c2.jpeg 2x" data-dominant-color="E2E2E2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">example_nn</span><span class="informations">1024×653 165 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The architecture could be possibly implemented by means of LSTM network. However, the plan is to use many blocks in training time and just a single MLP block upon testing (prediction). The main requirement here: all MLP blocks must be identical. So, the questions:</p>
<ol>
<li>How can I enforce all the blocks to have the same weights and biases during the training optimisation? One possible solution could be as follows: make a single step in gradient descent, then replace weights and biases by their means across all the blocks. How can I implement this procedure? Any better solution?</li>
<li>How can I connect the outputs of the previous blocks to the inputs of the subsequent ones bypassing certain entries in the input data matrix?</li>
<li>How can I simultaneously feed the outputs of the blocks into a loss function: Loss = (z1 - y1)**2 + (z2 - y2)**2 + (z3 - y3)**2, where y1, y2, y3 stand for the ground-truth values (labels).</li>
</ol>
<p>I would greatly appreciate it, if someone kindly gives me a suggestion.</p>

ptrblck: <ol>
<li>
<p>You could reuse the same module. E.g. if the block is a linear layer, you could call the same layer in each step.</p>
</li>
<li>
<p>Try to create new tensors via <code>torch.cat</code> using the new tensor entries and (some of) the output of the previous forward pass.</p>
</li>
<li>
<p>You could directly use the method and call <code>loss.backward()</code> after summing all losses.</p>
</li>
</ol>

Albert65: <p>Thank you very much. Your advice is really helpful.</p>