How can I run 2 parallel CNN on same data then combine outputs at FC layer?

Wafaa_Wardah: <p>I want to have a CNN architecture with the below pipeline:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b00a1b1b3e34bdc8be4471a57c20e658e7d5f91.png" data-download-href="https://discuss.pytorch.org/uploads/default/9b00a1b1b3e34bdc8be4471a57c20e658e7d5f91" title="DeepInsightArchitecture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9b00a1b1b3e34bdc8be4471a57c20e658e7d5f91_2_690x251.png" alt="DeepInsightArchitecture" data-base62-sha1="m7dmohdw5uDHoO8kMf3kkBbcByh" width="690" height="251" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9b00a1b1b3e34bdc8be4471a57c20e658e7d5f91_2_690x251.png, https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b00a1b1b3e34bdc8be4471a57c20e658e7d5f91.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b00a1b1b3e34bdc8be4471a57c20e658e7d5f91.png 2x" data-dominant-color="F0F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">DeepInsightArchitecture</span><span class="informations">694×253 80.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’ve done normal CNN using PyTorch before. Can someone please suggest if there’s a feature in PyTorch that let’s me easily split as above?</p>
<p>Thanks so much in advance.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a> Please help <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=12" title=":pensive:" class="emoji" alt=":pensive:" loading="lazy" width="20" height="20"></p>

albanD: <p>Hi,</p>
<p>If you have two cnns, you can simply do:</p>
<pre><code class="lang-python">inp = # Your input
out1 = cnn1(inp)
out2 = cnn2(inp)
full_out = torch.cat([out1, ou2], dim=1) # dim=1 is the channel dimension for 2D images
full_out = full_out.view(full_out.size(0), -1) # Linearize for the FC
res = fc(full_out)
</code></pre>
<p>Does that work for you?</p>

jmandivarapu1: <p>How to back propagate the loss?</p>

albanD: <p>It will back propagate automatically. Nothing special needs to be done:</p>
<pre><code class="lang-python">loss = crit(res, label)
loss.backward()
</code></pre>
<p>Then it’s your choice to have one optimizer for both nets, or one per net.</p>

jmandivarapu1: <p>Thank you &amp; I Assume it’s like this</p>
<p>optimizer = optim.SGD([p for p in model1.parameters()] +<br>
[p for p in model2.parameters()]),<br>
lr=0.001, momentum=0.9))</p>

albanD: <p>if you want both of them in the same optimizer yes.</p>

keyu_Chen: <p>The optimizer you defined seems only contain the parameters of Conv layers, what about the FC layers?</p>

JoshD: <p>Hi,</p>
<p>I wonder if the two CNN are running parallelly. If they run sequentially, it could be inefficient.</p>

albanD: <p>Depends on how you implement it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
But in general, each one use all the parallelism available on the current machine by default. So you won’t get any benefit by running the two in parallel on that machine as they will fight for resources.</p>

JoshD: <aside class="quote no-group" data-username="albanD" data-post="9" data-topic="66551">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>parallelism</p>
</blockquote>
</aside>
<p>What is the parallelism? So if I have multiple CNN branches and each one runs independently, I only need to write them down one by one and need not tell PyTorch these CNNs can run parallelly in some way?</p>

albanD: <p>What I mean is that even simple ops like one conv will already be parallelized if it is big enough.</p>

JoshD: <p>I see. My concern is regarding small conv. I heard that if we compare the runtimes of a big Conv layer and a series of cascaded small Conv layers, we will find that the cascaded small Conv layers takes more times to run even if the two settings have the same number of floating point operations. This is because small Conv layers cannot make full use of the GPU.</p>
<p>So I wonder if PyTorch has a mechanism to parallelize the small Conv layers that runs independently to accelerate it.</p>

albanD: <p>The problem is that the GPU itself is not built to run independent tasks in parallel. So you won’t be able to get the same speed with many small ops as with a single big one.</p>
<p>If you run on CPU, you can use multithreading as you would do for any workload to parallelize your execution.</p>