How and why are the input/output channels changed in CNNs?

make1234: <p>Hello,</p>
<p>Perhaps my question is slightly too theoretic for this forum - I know that input channels for eg. a <code>nn.Conv2d</code> layer are usually 3 when we have a coloured image - depicted as RGB. So if we specify <code>nn.Con2d(in_channels, out_channels,...)</code> we tell the CNN to set the number of these dimensions. Why would one change the number of the channels when an image is naturally made up of red blue and green?</p>

KFrank: <p>Hi Make!</p>
<aside class="quote no-group" data-username="make1234" data-post="1" data-topic="218139">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/make1234/48/74826_2.png" class="avatar"> make1234:</div>
<blockquote>
<p>Why would one change the number of the channels when an image is naturally made up of red blue and green?</p>
</blockquote>
</aside>
<p>The short answer is that doing so has been found empirically to be useful (admittedly,<br>
not a very satisfying answer).</p>
<p>With the disclaimer that intuition – even that of experts – about why various neural-network<br>
techniques work is notoriously unreliable, let me give you my thoughts:</p>
<p>Let’s consider a series of 3x3 convolutions (with padding, so that the spatial size of the<br>
image doesn’t change).  Let’s start with a three-channel ((RBG) input, and imagine that<br>
the number of channels is increased to, say, 8 and then 16 and then 32 and so on.</p>
<p>Consider a specific pixel location, more or less in the middle of the image.  After the first<br>
convolution, the pixel at that location depends on the 3x3 square of pixels from the<br>
original image that surrounds that specific location.  (All 8 channel values for that pixel<br>
after the first convolution depend on all 3 channel values for all 9 of the surrounding<br>
pixels in the original image.)  After the second convolution, that pixel depends on the<br>
5x5 square of surrounding pixels (again, with all channels depending on all channels).</p>
<p>Think of the channels as being “features” rather than colors.  Those features may or<br>
may not encode color information (depending on how the network is trained).  Speaking<br>
figuratively, if the original pixel in question is blue and one of the immediately adjacent<br>
pixels is, say, green, one of the channels after the first convolution might “detect” this<br>
as a “blue-green-edge” feature, and after another couple of convolutions, this may<br>
lead to an “iris” feature that then leads to an “eye” feature.</p>
<p>So the series of convolutions with increasing numbers of <code>out_channels</code> is converting<br>
spatial information (for which color – encoded in the three original color channels – may<br>
be important) into features (for which color may or may not still be important).</p>
<p>That is, we are increasing the number of channels so that the network can process spatial<br>
information into feature information.</p>
<p>Now consider a commonplace architecture where convolutions are followed by<br>
downsampling, say a series of 3x3 convolutions interleaved with 2x2 max-pools.  The<br>
network is now dumping the spatial information into features attached to a shrinking<br>
number of pixels, leading, potentially, to an output image that consists of a <em>single</em><br>
pixel – that is, <em>no</em> spatial information – with all of the features encoded in the channels<br>
dimension being derived from the entire image.</p>
<p>In such a situation, one could well imagine that none of those features directly contain<br>
any color information.  (Of course, if you trained a network to distinguish red-tinged<br>
images from blue-tinged images, you would likely have a red-vs-blue feature encoded<br>
somewhere in the channels dimension.)</p>
<p>Best.</p>
<p>K. Frank</p>

make1234: <p>Thanks for your reply!</p>

J_Johnson: <p>Simply put: feature extraction. Suppose we are trying to identify cat vs. dog. Ears that point up would be more indicative of a cat. While ears that droop would be more dog-like. So we use kernels, say 3x3 that take each 3x3 square in the image and matmul it, to derive a new pixel for the output of that layer, each kernel corresponding to a separate output channel. That’s applied at each point.</p>
<p>Some of those kernels may excel at exaggerating droopy ears while others at exaggerating pointed ears. Same for whiskers, face shape, and anything else that might help us identify a dog or cat.</p>
<p>Finally, those channels get funneled into probabilities for each feature. And then the linear layers provide the logical component: If “pointy ears”, “triangular face”, and “has whiskers” channels score high, then it’s probably a cat.</p>

make1234: <p>I have another question - say I have a RGB image and 3 input channels initially. when I want to expand these 3 channels to 5 how are they computed exactly? Using 5 different kernels on 3 in-channels? how do we decide which of the 3 in-channels to use more than once?</p>

J_Johnson: <p>This provides a good visual gif partway down for 3 channels to 1 output channel. Just do the same with additional kernels for multiple channels.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://saturncloud.io/blog/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way/">
  <header class="source">
      <img src="https://saturncloud.io/favicons/favicon.ico" class="site-icon" width="48" height="48">

      <a href="https://saturncloud.io/blog/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way/" target="_blank" rel="noopener nofollow ugc" title="12:00AM - 15 December 2018">saturncloud.io – 15 Dec 18</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/233;"><img src="https://saturncloud.io/images/blog/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way.webp" class="thumbnail" width="690" height="233"></div>

<h3><a href="https://saturncloud.io/blog/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way/" target="_blank" rel="noopener nofollow ugc">A Guide to Convolutional Neural Networks — the ELI5 way | Saturn Cloud Blog</a></h3>

  <p>Artificial Intelligence has been witnessing monumental growth in bridging the gap between the capabilities of humans and machines. Researchers and enthusiasts alike, work on numerous aspects of the field to make amazing things happen. One of many...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>