How to build a network consisting of several parallelly connected subnetworks

zjsong: <p>Hi, all,</p>
<p>Recently, I want to use PyTorch to build a slightly special network with the following structure:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/c7a32d74dbf9dd13c0011349d0bbc4d9088cea0f.png" data-download-href="https://discuss.pytorch.org/uploads/default/c7a32d74dbf9dd13c0011349d0bbc4d9088cea0f" title="network_structure"><img src="https://discuss.pytorch.org/uploads/default/original/2X/c/c7a32d74dbf9dd13c0011349d0bbc4d9088cea0f.png" alt="network_structure" data-base62-sha1="su4CCzJg1hDOd2dzz1JRT2rdmmr" width="690" height="388" data-dominant-color="EBF1EE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">network_structure</span><span class="informations">1280×720 27.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In the above network, the green boxes correspond to known variables, i.e., input vector <strong>x</strong> and m label vectors <strong>y</strong>^(1), …, <strong>y</strong>^(m). Here each <strong>y</strong>^(i) is a one-hot vector with value (1,0) or (0,1). And the remaining gray boxes indicate hidden representations (e.g., <strong>z</strong>^(1,1) and <strong>z</strong>^(2,m)).</p>
<p>My question is for a given positive integer <code>m&gt;1</code>, how to implement such a network in <code>nn.Module</code> class? I guess maybe I need to use <code>nn.ModuleList()</code> and the <code>for</code> loop.</p>
<p>Could anyone please give a further suggestion or comment? Thanks in advance.</p>

zjsong: <p>P.S.: all hidden representations in the same layer have the same size, e.g., <strong>z</strong>^(1,i).size()=(100,1) and <strong>z</strong>^(2,j).size()=(10,1) for all i and j.</p>

jpeg729: <p>A naive approach would indeed be to construct <code>m</code> using <code>nn.ModuleList()</code> and a for loop.<br>
But I think we parallelise the whole lot. The following idea should work, but I haven’t checked all the details.</p>
<p>The input <code>x</code> is shared by all subnetworks, so a single Linear layer of size <code>m*100</code> followed by separating the result into <code>m</code> parts will work identically to <code>m</code> separate Linear layers of size <code>100</code>.</p>
<pre><code class="lang-auto">self.x_to_z1 = nn.Sequential(
    nn.Linear(x_size, 100*m),
    nn.ReLU(),
)
</code></pre>
<p>In <code>forward()</code> we could follow this by <code>.view(batch_size, m, 100)</code> to separate the z^(1,i). Or we could use <code>.view(batch_size, 100, 1)</code> to add a spatial dimension of size 1, and follow up with a grouped convolution.</p>
<pre><code class="lang-auto">self.z1_to_z2 = nn.Sequential(
    nn.Conv1d(100*m, 10*m, kernel_size=1, groups=m),
    # the input is grouped into groups of 100 channels, 
    # and each group is used to produce 10 channels of output
    nn.ReLU(),
)
</code></pre>
<p>And then another grouped convolution to produce the required number of outputs, i.e. 2 per y^i, if I have understood correctly.</p>
<pre><code class="lang-auto">self.z2_to_y = nn.Conv1d(10*m, 2*m, kernel_size=1, groups=m)
</code></pre>
<p>And finally you could use <code>.view(batch_size, m, 2)</code> followed by <code>.split(1, dim=1)</code> to split the output into a list of <code>m</code> tensors of size 2. But I suspect you could parallelise the loss function too.</p>

zjsong: <p>Dear <a class="mention" href="/u/jpeg729">@jpeg729</a>,</p>
<p>Thank you so much for your suggestion. I think that your ideas of grouping hidden representations and using 1D convolution are pretty brilliant!!! And I will have a try accordingly.<img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=5" title=":grin:" class="emoji" alt=":grin:"></p>
<p>Regards,<br>
Zengjie Song</p>

jpeg729: <p>I have to admit that I would have probably gone for the naive approach, but I took the question as a challenge and had that idea.</p>
<p>One detail that I am not sure of is how the grouped conv groups the channels. Does a <code>Conv1d(100*m, 10*m, 1, groups=m)</code> use the first 100 channels input to produce the first 10 channels output? Or do the first m channels of input &amp; output all belong to different groups? I suspect the former, but I don’t know.</p>

zjsong: <p>Maybe all of these queries would be solved by doing some experiments. I would like to have a try first, and then give the validation results and discoveries. Thanks.</p>

zjsong: <p>Sorry for the delayed reply. After doing some experiments, I found that the alternative method proposed by <a class="mention" href="/u/jpeg729">@jpeg729</a> is feasible.</p>

semperDM: <p>Hello,</p>
<p>I was looking for clarification on this topic, as I think I am trying the same thing.</p>
<p>The goal here was to construct 3 networks, that all used the same input data, had the same hidden layer structure, but had 3 different outputs correct?</p>
<p>Thank you</p>

zjsong: <p>Yes, that is almost the case discussed here.</p>
<p>Conceptually we expect to obtain a network consisting of 3 (or m) sub-networks that have the same architecture but with different connection weights.</p>

semperDM: <p>Did you find that there were issues with overtraining the network?  For example, say that y1 was very easy to learn, but y3 was hard to learn.  Was there a single loss function being minimized, or was there a separate loss function for each subnetwork?</p>
<p>If there was a single loss function, I worry then that y1 being easy to learn would result in it being overtrained, due to the difficulty in learning y3.</p>

zjsong: <p>I think that whether overtraining happens depends on many factors, e.g., network architecture, loss function, data, etc.</p>
<p>To address the case that y1 is easy to learn compared with others, maybe we can construct a shared network backbone, and then adding specific head (or sub-network) for each yi. So each sub-network would be learnt separately according to the loss part containing yi.</p>