What is a neuron in a convolutional layer of CNN?

AjeelAhmed1998: <p>I realize that this is not exactly on topic since this is a more conceptual question but I was hoping to get some insight.</p>
<p>From the famous CS231n <a href="https://cs231n.github.io/convolutional-networks/" rel="noopener nofollow ugc">lecture notes</a>:</p>
<blockquote>
<p>Local Connectivity. <strong>When dealing with high-dimensional inputs such as images, as we saw above it is impractical to connect neurons to all neurons in the previous volume. Instead, we will connect each neuron to only a local region of the input volume</strong>. The spatial extent of this connectivity is a hyperparameter called the receptive field of the neuron (equivalently this is the filter size). The extent of the connectivity along the depth axis is always equal to the depth of the input volume. It is important to emphasize again this asymmetry in how we treat the spatial dimensions (width and height) and the depth dimension: The connections are local in 2D space (along width and height), but always full along the entire depth of the input volume.</p>
<p>Example 1. For example, suppose that the input volume has size [32x32x3], (e.g. an RGB CIFAR-10 image). <strong>If the receptive field (or the filter size) is 5x5, then each neuron in the Conv Layer will have weights to a [5x5x3] region in the input volume</strong>, for a total of 5x5x3 = 75 weights (and +1 bias parameter). Notice that the extent of the connectivity along the depth axis must be 3, since this is the depth of the input volume.</p>
</blockquote>
<p>What is the neuron here? There are only kernels in a convolutional layer so why bring neurons into this?</p>
<p>Moreover what does it mean to connect each neuron to only a local region of the input volume? When we convolve we are not looking at a single small region but we in fact go over the whole image by sliding our kernel over the input image.</p>
<p>Finally is filter size and receptive field the same thing? The lecture notes seem to indicate they are.</p>

KFrank: <p>Hi Ajeel!</p>
<aside class="quote no-group" data-username="AjeelAhmed1998" data-post="1" data-topic="159111" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ajeelahmed1998/48/52089_2.png" class="avatar"> AjeelAhmed1998:</div>
<blockquote>
<p>What is the neuron here?</p>
</blockquote>
</aside>
<p>The short story is that “neuron” is not always well defined.</p>
<ol>
<li>
<p>It’s a jargon term.</p>
</li>
<li>
<p>The analogy between machine-learning neural networks and actual<br>
neurons and brains is quite weak and often stretched beyond the<br>
breaking point.</p>
</li>
<li>
<p>Often, a lot of the neural-network literature, including refereed papers,<br>
uses language unhelpfully loosely, so even if there were an agreed-upon<br>
definition for “neuron,” you wouldn’t be able to  count on people using<br>
the term consistently with the definition.</p>
</li>
</ol>
<p>Having said that:</p>
<p>Neural networks were inspired by biological neurons (but to say that they<br>
are modelled on biological neurons would be an overstatement).</p>
<p>Here’s a cartoon view of a neuron (that is connected to other neurons):</p>
<p>It has a main cell body, a bunch of “tentacles” called dendrites that<br>
receive signals from other neurons, and an axon that carries the signal<br>
somewhere else that ends in some more tentacles (“axon terminals”) that<br>
transmit the signal to the dendrites of other neurons through synapses.</p>
<p>The basic idea is that the connections between two neurons can have<br>
different strengths and can be excitatory or inhibitory.  Because a neuron<br>
receives inputs from multiple other neurons, the simplified picture is that<br>
if the excitatory inputs are stronger, in aggregate, than the inhibitory inputs,<br>
the neuron will “fire” and may fire more or less strongly.</p>
<p>In pytorch language, consider:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.nn.Sequential (
    torch.nn.Linear (10, 10),
    torch.nn.ReLU(),
    torch.nn.Linear (10, 2),
    torch.nn.ReLU()
)
</code></pre>
<p>Focus on the first <code>ReLU</code> and the second <code>Linear</code> (and ignore batch and<br>
other dimensions):</p>
<p>For each of the two outputs of the second <code>Linear</code>, there are ten inputs.<br>
You can think of the <code>weight</code> property of the second <code>Linear</code> as being the<br>
strengths of the synaptic connections between ten input neurons and two<br>
output neurons.  Because the weighted inputs are summed in the matrix<br>
multiplication inside of the <code>Linear</code>, you can think of this step as aggregating<br>
together the signals from the input neurons, with positive weights<br>
corresponding to excitatory synaptic connections and negative weights<br>
corresponding to inhibitory connections.  Then the <code>bias</code> and subsequent<br>
non-linear activation – in this case, the second <code>ReLU</code> – determine whether<br>
and how strongly the output neuron will fire.</p>
<p>Let’s try to push the biological-neuron analogy as far as we can. Where,<br>
exactly, is the neuron?  The second <code>Linear</code>, and therefore the second<br>
<code>ReLU</code>, have (ignoring batch and other dimensions) two scalar outputs.<br>
I like to think of a single scalar output of the <code>ReLU</code> as corresponding<br>
to the axon of a neuron.  The <code>ReLU</code> itself and the <code>bias</code> and the <code>weight</code><br>
of the <code>Linear</code> are spread in some fashion over the cell body and the<br>
dendritic synaptic clefts of the <em>two</em> neurons associated with the second<br>
<code>Linear</code>.</p>
<p>That covers the dendrites, cell body, and axon of our biological neuron.<br>
But where, then, are the axon terminals of our neuron?  I would say that<br>
they are in the <code>weight</code> of some subsequent <code>Linear</code> layer.</p>
<p>So if you were to ask me what a neuron is in a neural network, the second<br>
best answer I could give would be to say that a single output of a non-linear<br>
activation layer is a neuron – or, more precisely, is the axon of a neuron<br>
(with the rest of the neuron being distributed over other pieces of the<br>
network, as outlined above).  (My best answer would be to say don’t take<br>
the biological-neuron analogy too seriously.)</p>
<aside class="quote no-group">
<blockquote>
<p>There are only kernels in a convolutional layer so why bring neurons into this?</p>
</blockquote>
</aside>
<p>Coming back to your original question, where in a <em>convolutional</em> layer<br>
is a neuron?  This gets dicier still.</p>
<p>Consider early biological image processing:  Illustratively speaking, in<br>
the retina (or perhaps early in the visual cortex), there is some cluster<br>
of neurons in the lower-left corner of the visual field that has (perhaps)<br>
connections that implement an edge detector, and a similar “edge-detector”<br>
cluster likely exists in the upper-right of the visual field, as well.</p>
<p>In a neural network the upstream convolutions are thought to “learn” such<br>
low-level features, so one imagines that some upstream convolution learns<br>
edge-detection weights.  This one convolutional kernel would perform edge<br>
detection for the entire input image, including (among other locations)<br>
the lower left and upper right.</p>
<p>Thus, the difference between the convolutional layer and the biological<br>
system is that there is only one convolution kernel that gets slid over<br>
(convolved with) the input image, while the retina has visual-processing<br>
neurons across its entire back surface, and different sets of such neurons<br>
perform early visual processing (e.g., edge detection) for different parts of<br>
the visual field (e.g., the lower left vs. the upper right).</p>
<p>So if I had to identify a “neuron” in a convolutional layer, I would note that<br>
the output of the convolutional layer (including its non-linear activation), is<br>
a whole “image,” not just a kernel-sized patch, and then say that each scalar<br>
element of this “image” is the axon of a neuron.</p>
<p>So far, so good.  But where are that neuron’s dendrites?  The best I can say<br>
is that the convolution kernel (in part) represents a bunch of virtual sets of<br>
dendrites, one set for each axon, as the kernel is slid over the input image.</p>
<aside class="quote no-group">
<blockquote>
<p>Moreover what does it mean to connect each neuron to only a local region of the input volume?</p>
</blockquote>
</aside>
<p>Based on my discussion above, if we say that (an axon of) a neuron is a<br>
single element of the output “image” tensor of the convolutional layer<br>
(including its associated non-linear activation), then yes, that neuron only<br>
depends on a local region of the input image.</p>
<aside class="quote no-group">
<blockquote>
<p>When we convolve we are not looking at a single small region but we in fact go over the whole image by sliding our kernel over the input image.</p>
</blockquote>
</aside>
<p>As you slide the convolution kernel along the input image, it’s only one<br>
position of the kernel – and hence only one local region of the input<br>
image – that contributes to a particular single element of the output<br>
“image” tensor.</p>
<aside class="quote no-group">
<blockquote>
<p>Finally is filter size and receptive field the same thing?</p>
</blockquote>
</aside>
<p>Yes, the receptive field of a “neuron” (as I define it above) is just that<br>
region of the input image that contributes to the single element of the<br>
output tensor that I am calling the neuron.</p>
<p>If we have, say, a 3x3 filter (convolution kernel), then when the filter is in<br>
the location, as it is being slid along, that corresponds to that particular<br>
element of the output tensor, the 3x3 region of the input image that the<br>
filter is “on top of” is the only part of the input image (assuming no convolution<br>
dilations) that affects that element of the output tensor.  That 3x3 region of<br>
the input image is the “receptive field” of the neuron (that is, of that single<br>
element of the output tensor).  If we had, say, a 5x7 kernel, the receptive<br>
field (of a particular neuron) would be a (particular) 5x7 region of the input<br>
image.</p>
<p>But, yeah, I’m not a big fan of saying that neural networks have well-defined<br>
neurons in them, and even less a fan when speaking of convolutional layers.</p>
<p>Best.</p>
<p>K. Frank</p>

AjeelAhmed1998: <p>Hey <a class="mention" href="/u/kfrank">@KFrank</a> thank you so much for such a detailed explanation, I do agree the actual neuron analogies is a bit too much to understand at times, it is why I was having trouble in the first place but if you go all in and start looking at <em>everything</em> in terms of neurons, it starts making a lot more sense. What really helped me was watching Andrej Karpathy’s lectures from CS231n.</p>
<p>Thank you so much once again <a class="mention" href="/u/kfrank">@KFrank</a>!</p>