How do the top and bottom layers of AlexNet communicate?

username: minimafinder
cooked: <p>I am trying to re-implement <a href="https://papers.nips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf" rel="noopener nofollow ugc">Krizhevsky et al. (2012)</a> using PyTorch, and <strong>I am confused how precisely the second and the third convolutional layers of the AlexNet model communicate</strong> (and same for inputs from the fifth to the sixth layer, and from the sixth to the seventh layer, although I am omitting this in my question here).</p>
<p>In the figure below, there are two “filters” that pass the output from the top half to next top half but also the bottom half. Similarly, the bottom half has two “filters” that pass the output to the next bottom and top half.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d09c5a81af10f7ac275577cb9c7f6379957c5a0e.png" data-download-href="https://discuss.pytorch.org/uploads/default/d09c5a81af10f7ac275577cb9c7f6379957c5a0e" title="alexnet_2nd_3rd_layers"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d09c5a81af10f7ac275577cb9c7f6379957c5a0e_2_488x500.png" alt="alexnet_2nd_3rd_layers" data-base62-sha1="tLsiAjLay4Rieq9FBwRsNjMVgjs" width="488" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d09c5a81af10f7ac275577cb9c7f6379957c5a0e_2_488x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d09c5a81af10f7ac275577cb9c7f6379957c5a0e_2_732x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/0/d09c5a81af10f7ac275577cb9c7f6379957c5a0e.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">alexnet_2nd_3rd_layers</span><span class="informations">790×808 63.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><em><strong>My main question:</strong></em> <strong>How are the outputs of the second layer being passed to the third layer?</strong></p>
<p>I read the paper, and unless I missed something, it seems that the authors didn’t precisely outline how the outputs are being passed from the second to the third layer. I skimmed through a bunch of blogposts and git repositories, and most descriptions are high-level and most implementations don’t seem to split the model between two GPUs.</p>
<p>The most relevant thing I could find was the following sentence from the <a href="https://github.com/akrizhevsky/cuda-convnet2/blob/wiki/MultiGPU.md" rel="noopener nofollow ugc">convnet2 readme</a>:</p>
<blockquote>
<p>Here, the layers conv2a and conv2b take both conv1a and conv1b as inputs. An implicit copy operation is performed in order to get the output of conv1a into the input of conv2b, as well as the output of conv1b into the input of conv2a.</p>
</blockquote>
<p>My best guess is that the <code>out_channels</code> parameter in the second layers should actually be 64 instead of 128, and then the output from both the top and bottom layer should be concatenated as <code>torch.cat([output_from_top_half, output_from_bottom_half], dim=1)</code> and passed to the top and bottom half of the third layer. I am unsure if my understanding here is correct, though.</p>
<p>I am confused what’s the correct interpretation here, and would appreciate any helpful pointers!</p>