Some kernels entirely zero in cnn?

jayz: <p>Hey,</p>
<p>I have trained a CNN with many feature maps. When inspecting the weights of the kernels of the individual conv2d layers, i figured that in the first layer right after the raw input there are two (of 16) kernel maps all zero. The remaining kernel maps through all layers are populated with non-zero values. Is it a common thing to happen? The model seems to fit my training data, so far, so I guess this is not a big problem?</p>
<p>Thanks!</p>
<p>Best, JZ</p>

KFrank: <p>Hi Jay!</p>
<aside class="quote no-group" data-username="jayz" data-post="1" data-topic="146735" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jayz/48/46742_2.png" class="avatar"> jayz:</div>
<blockquote>
<p>in the first layer right after the raw input there are two (of 16) kernel maps all zero.</p>
</blockquote>
</aside>
<p>I have never seen this happen, and I don’t believe that it is common,<br>
so you should definitely check your code for a bug or design error.</p>
<p>I think it could plausibly happen, though.  Here’s a not-too-far-fetched<br>
scenario that I could imagine:</p>
<p>Weight decay, if you are using it, would tend to drive your weights toward<br>
zero.  If the gradients for some of your weights end up being zero, there<br>
will be nothing to push the weights back away from zero.</p>
<p>If your forward pass goes through the flat part (<code>x &lt; 0</code>) of a <code>ReLU</code> or the<br>
saturated part (large positive or negative value) of a <code>Sigmoid</code>, you will<br>
get zero gradients on the backward pass.</p>
<p>Some things to check:</p>
<p>Do your weights start off zero immediately after (random) initialization?<br>
(Presumably not.)</p>
<p>Do your gradients become zero at some point?  If so, can you identify<br>
why?</p>
<p>Do your weights decrease toward zero before becoming zero?  Are<br>
gradients small or zero while this is happening?  This would be consistent<br>
with weight decay and zero gradients driving the weights to zero.</p>
<p>If you retrain the network from scratch (with differing random initialization),<br>
do the same kernels become zero every time, or do different sets of<br>
kernels end up zero?</p>
<p>Best.</p>
<p>K. Frank</p>

jayz: <p>Hello K,</p>
<p>thanks for the reply!</p>
<p>Yes, I was in fact using weight decay in that particular instance. I am just training several models with and without different levels of weight decay, and will see whether the wd is the cause of this, as soon as the training is done.</p>
<p>Also, yes, I am using ReLU in between the conv-layers, followed by Batch Normalization and Dropout. But aint that a common thing to use ReLU plus weight decay in conv networks?</p>
<p>Will check the initial weights, and their values within the first few iterations, as well as the gradients. Do you have a good procedure for monitoring the gradients within the network? I guess its done via the backward_hook method, right? I would then collect multiple gradients every time a batch has been processed and compute the average of their elementwise squared entries, or sth like that?</p>
<p>Also, I could probably inspect the relu activation output, right away, without inspecting the gradients, right? If the ReLU outputs all-zero multiple times, this could be the cause?</p>
<p>Thanks!</p>
<p>Best, JZ</p>

zetyquickly: <p>Have you tried to initialize weights with Kaiming or Xavier initializer? May it be the case that you have zeroed weights from the start and further?</p>

jayz: <p>Hey,</p>
<p>thanks for the reply!<br>
yes, I am initializing all the conv layers with xavier. For the batch normalization layers, I leave weights = all 1 and bias = all 0. Is that ok, or should they also be randomized?</p>
<p>Best, JZ</p>

zetyquickly: <p>Not sure whether BatchNorm has weights and biases</p>