Grouping some features before going to fully connected layers

cosmicnet: <p>Hello,</p>
<p>I found something kinda similar here:</p><aside class="quote quote-modified" data-post="1" data-topic="45517">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/animesh_kumar_paul/48/11990_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/implement-selected-sparse-connected-neural-network/45517">Implement Selected Sparse connected neural network</a> 
  </div>
  <blockquote>
    I am trying to implement the following general NN model (Not CNN) using Pytorch. 
 <a class="lightbox" href="https://lh6.googleusercontent.com/Boki3HCcAbmFrCHLfyLEOUaQOQlTffYSGi5iAzGyne_dG3YUbonkl5O8n0iUcWSDQQHdxiZON-l2SYksF8L2RI5hjO4MQTRGQPdbtJqn0Z1EIQAf8YnhSvVNnqNwJXLUGpoQRMvowvQ" title="Boki3HCcAbmFrCHLfyLEOUaQOQlTffYSGi5iAzGyne_dG3YUbonkl5O8n0iUcWSDQQHdxiZON-l2SYksF8L2RI5hjO4MQTRGQPdbtJqn0Z1EIQAf8YnhSvVNnqNwJXLUGpoQRMvowvQ" rel="noopener nofollow ugc">[|710px;x540px;]</a> 
Here, 3rd, 4th, 5th layers are fully connected-- and Network 1,2, 3 itself are fully connected but they are not fully connected to each other. 
I don’t know how to implement this kind of selected (Not Random) sparse connection in Pytorch. 
Any help/comments on this are much appreciated.
  </blockquote>
</aside>

<p>But the conversation doesn’t appear to quite cover what I’m looking to do.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d05334871c725f2958405ade874b336d7459014d.png" data-download-href="https://discuss.pytorch.org/uploads/default/d05334871c725f2958405ade874b336d7459014d" title="grouped_features_network.drawio"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d05334871c725f2958405ade874b336d7459014d.png" alt="grouped_features_network.drawio" data-base62-sha1="tIVzY4sRzSYDlj3fNvPDWsr7gFD" width="438" height="500" data-dominant-color="EEEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grouped_features_network.drawio</span><span class="informations">501×571 31.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I want to produce a network like the diagram above. Some features will be grouped together before going to the fully connected layers (group 1), there maybe be some overlap between groups (2 and 3), whereas other features will not be grouped at all and just feed directly into the fully connected layers. The diagram shows only one fully connected layer as an example but there would actually be several.</p>
<p>I hope this makes sense.</p>
<p>I want to do this as we found that one engineered feature we had, which was made from a linear regression of a group of related features, was very effective and I’d like to generalise that into the network itself and try for other features we think are related.</p>

cosmicnet: <p>I’m not sure whether there are no replies due to this not being possible, or that I should just get on trying to build it…</p>

AlphaBetaGamma96: <p>Can you share more than just a figure? Perhaps what shape each group is? Because it seems like you have 3 feed-foward networks for each group that map some inputs features to 3 different scalars, then paass those scalars (with the ungrouped scalars) into another feed-forward network which maps them all to a single scalar. So, more detail on the input data (i.e. batch size, input shapes etc…) will make it easier for people to debug.</p>

cosmicnet: <p>Hi,</p>
<p>What you’ve described is accurate. It’s not a request for debugging, more a request as to whether such a network is feasible and any pointers before I expel the effort trying to make it.</p>
<p>As for the data, I’m currently working with multivariate tabular data. For the purposes of the illustration each leftmost node is a single numerical feature, and the target is a regression. How does batch size factor in?</p>

AlphaBetaGamma96: <p>You could just create those “group” networks via <code>nn.Module</code> objects then call them, and pass their outputs to the final network on the right-hand-side of the figure. So, something like,</p>
<pre><code class="lang-auto">out1 = group1net(in1) #returns batch of scalars
out2 = group2net(in2) #returns batch of scalars
out3 = group3net(in3) #returns batch of scalars
#ungroup is shape [B,2]
x = torch.cat([out1, out2, out3, ungroup], dim=-1) #concatenate into single input tensor

output = finalnet(x) #simply pass to single-layer network on the right. (outputs scalar)
</code></pre>
<p>Given group2 and group3 share an input feature you can simple handle that when you create the <code>in2</code> and <code>in3</code> input Tensors, and let your network handle the rest.</p>
<aside class="quote no-group" data-username="cosmicnet" data-post="4" data-topic="162444">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/b9e5f3/48.png" class="avatar"> cosmicnet:</div>
<blockquote>
<p>How does batch size factor in?</p>
</blockquote>
</aside>
<p>Make sure the batch size is consistent across all inputs features!</p>

cosmicnet: <p>Thanks for your advice. I’ll give this a go and see what kinds of results I get.</p>