What kind of filters are used with Conv2d?

modeler: <p>I don’t think I found any info on this online but what are the default filters used by Conv2d?</p>

aditya98ak: <p>I think filters are initialized with random values and they are updated via backpropagation.</p>

modeler: <p>Wait the filter values are updated, meaning that they likely change, every epoch?</p>

aditya98ak: <p>As per my intutions, they should update every time when we do loss.backward().<br>
Am I in the right directions, experts?</p>

ptrblck: <p><code>loss.backward()</code> calculates all gradients in the current computation graph (including the filter weight gradients). The weights are updated if you call <code>optimizer.step()</code> (and passed these parameters to the optimizer before) or update the weights manually using the gradients.<br>
<a class="mention" href="/u/modeler">@modeler</a> in fact the weights change in every iteration if the gradients are non-zero.</p>

aditya98ak: <p>oh, yes! sorry for the misconfusion I created <a class="mention" href="/u/modeler">@modeler</a>.<br>
Thanks for the answer <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>

modeler: <p>What is the reasoning for a randomized initialization as opposed to some non-zero constant initialization?</p>

ptrblck: <p>I think symmetry breaking might be one reason, although the problem of equal outputs won’t be that serious like in linear layers.<br>
E.g. if you initialize a linear layer with some constant weight, each output will have the same value. Later in the backward pass this could create the same weight updates for each parameter etc. Each weight thus cannot learn anything “new”, and you would have a whole layer of a cloned neuron.<br>
Random initialization breaks this symmetry.</p>
<p>Also, I think another reason might be that your constant values might bias your model towards a particular solution, which might be useful, if you know what you are doing.</p>

tom: <p>In addition to Peter’s spot-on comments about symmetry breaking, there is a <a href="https://www.inference.vc/the-lottery-ticket-hypothesis/" rel="nofollow noopener">the lottery ticket hypothesis</a>, roughly speaking the theory that (overparametrised by traditional standards) NNs are “looking in many places of the parameter landscape, thereby picking up some useful ones”.</p>
<p>Weight initialization in particular is something  that has been identified as fairly important and I can recommend spending thought on it - PyTorch inherits the initializations mostly from Torch, and might not always reflect the latest advice of how to do it. Most stock modules have a method <code>reset_parameters</code> that has the default (e.g. do <code>?? torch.nn.Conv2d.reset_parameters</code> to see the source in IPython/Jupyter).<br>
In contrast to weight, bias can, in my experience, often just be zeroed.</p>
<p>Best regards</p>
<p>Thomas</p>

saba: <p>Hello ptrblck</p>
<p>I hope you are well. Sorry , I have a question about initialize the filters in pytorch. how I can specify them randomly from Gaussian distribution? are they from Gaussian distribution?</p>

ptrblck: <p>The filters in <code>nn.Conv2d</code> are initialized by <a href="https://github.com/pytorch/pytorch/blob/a561a8448b6b7bb8f6068178fe7bc9e2bf348304/torch/nn/modules/conv.py#L48">reset_parameters</a> as <a class="mention" href="/u/tom">@tom</a> mentioned.</p>
<p>To initialize them with a Gaussian distribution, you could use <a href="https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.normal_"><code>torch.nn.init.normal_</code></a>.</p>

saba: <p>i used Convd3. Where can I add this command in my code?</p>
<p>class ConvNetRedo1(nn.Module):<br>
def <strong>init</strong>(self,numf1,numf2,fz1,fz2,nn2,nn3): # numf1( nnumber of filters first layer)numf2(nnumber of filters first layer)),fz1 kernel size(),fz2,nn2,nn3<br>
super(ConvNetRedo1, self).<strong>init</strong>()<br>
self.numf1=numf1<br>
self.numf2=numf2<br>
self.fz1=fz1<br>
self.fz2=fz2<br>
self.nn2=nn2<br>
self.nn3=nn3<br>
self.layer1 = nn.Sequential(nn.Conv3d(1, self.numf1, kernel_size=self.fz1, stride=1, padding=2),nn.ReLU(),nn.MaxPool3d(kernel_size=2, stride=2))<br>
self.layer2 = nn.Sequential(nn.Conv3d(self.numf1,self.numf2, kernel_size=self.fz2, stride=1, padding=2),nn.ReLU(),nn.MaxPool3d(kernel_size=2, stride=2))</p>
<pre><code>    self.fc1 = nn.Linear(3072, self.nn2) ##3027
    self.drop_out1 = nn.Dropout(0.5)
    self.relu1 = nn.ReLU() 
    self.fc2 = nn.Linear( self.nn2, self.nn3) 
    self.drop_out2 = nn.Dropout(0.5)
    self.relu2 = nn.ReLU()              
    self.fc3 = nn.Linear( self.nn3, 1)</code></pre>

ptrblck: <p>You could write a <code>weight_init</code> function and call <code>model.apply</code> with it:</p>
<pre><code class="lang-python">def weights_init(m):
    with torch.no_grad():
        if isinstance(m, nn.Conv3d):
            torch.nn.init.normal_(m.weight)
            torch.nn.init.normal_(m.bias)

net.apply(weights_init)
</code></pre>

saba: <p>Is it correct to use??</p>
<p>def weights_init(m):<br>
with torch.no_grad():<br>
if isinstance(m, nn.Conv3d):<br>
torch.nn.init.normal_(m.weight)<br>
torch.nn.init.normal_(m.bias)</p>
<p>class ConvNetRedo1(nn.Module):<br>
def <strong>init</strong>(self,numf1,numf2,fz1,fz2,nn2,nn3):<br>
super(ConvNetRedo1, self).<strong>init</strong>()<br>
self.numf1=numf1<br>
self.numf2=numf2<br>
self.fz1=fz1<br>
self.fz2=fz2<br>
self.nn2=nn2<br>
self.nn3=nn3<br>
self.layer1 = nn.Sequential(nn.Conv3d(1, self.numf1, kernel_size=self.fz1, stride=1, padding=2),nn.ReLU(),nn.MaxPool3d(kernel_size=2, stride=2))</p>
<p>net.apply(weights_init)</p>
<pre><code>    self.layer2 = nn.Sequential(nn.Conv3d(self.numf1,self.numf2, kernel_size=self.fz2, stride=1, padding=2),nn.ReLU(),nn.MaxPool3d(kernel_size=2, stride=2))
</code></pre>
<p>net.apply(weights_init)</p>
<pre><code>    self.fc1 = nn.Linear(3072, self.nn2) ##3027
    self.drop_out1 = nn.Dropout(0.5)
    self.relu1 = nn.ReLU() 
    self.fc2 = nn.Linear( self.nn2, self.nn3) 
    self.drop_out2 = nn.Dropout(0.5)
    self.relu2 = nn.ReLU()                          
    self.fc3 = nn.Linear( self.nn3, 1) 

def forward(self, x):
  x=x.unsqueeze(1).float()
  out = self.layer1(x)
 out = self.layer2(out)
  out = out.view(out.size(0), -1)
    out = self.fc1(out)
    out=self.drop_out1(out)
    out=self.relu1(out)
   out = self.fc2(out)
    out=self.drop_out2(out)
    out=self.relu2(out)  
    out = self.fc3(out)
    return out</code></pre>

ptrblck: <p>Your format is a bit broken, but it seems you are trying to call <code>net.apply</code> inside the model definition?</p>
<p>PS: you can add code snippets by wrapping them in three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

saba: <p>Yes, I use ( <code>net.apply(weights_init)</code>) after each CNN layer.</p>
<pre><code class="lang-auto">    with torch.no_grad():
        if isinstance(m, nn.Conv3d):
            torch.nn.init.normal_(m.weight)
            torch.nn.init.normal_(m.bias)
class ConvNetRedo1(nn.Module):
    def __init__(self,numf1,numf2,fz1,fz2,nn2,nn3): # numf1( nnumber of filters first layer)numf2(nnumber of filters first layer)),fz1 kernel size(),fz2,nn2,nn3
        super(ConvNetRedo1, self).__init__()
        self.numf1=numf1
        self.numf2=numf2
        self.fz1=fz1
        self.fz2=fz2
        self.nn2=nn2
        self.nn3=nn3
self.layer1 = nn.Sequential(nn.Conv3d(1, self.numf1, kernel_size=self.fz1, stride=1, padding=2),nn.ReLU(),nn.MaxPool3d(kernel_size=2, stride=2))
net.apply(weights_init)
self.layer2 = nn.Sequential(nn.Conv3d(self.numf1,self.numf2, kernel_size=self.fz2, stride=1, padding=2),nn.ReLU(),nn.MaxPool3d(kernel_size=2, stride=2))
net.apply(weights_init)
  self.fc1 = nn.Linear(3072, self.nn2) 
  self.drop_out1 = nn.Dropout(0.5)
  self.relu1 = nn.ReLU() 
  self.fc2 = nn.Linear( self.nn2, self.nn3) 
    self.drop_out2 = nn.Dropout(0.5)
   self.relu2 = nn.ReLU()               
    self.fc3 = nn.Linear( self.nn3, 1) ```
Is it correct?</code></pre>

ptrblck: <p>Ah OK, you don’t need to call this method after each layer.<br>
Just initialize your model and call it once via <code>model.apply(weight_init)</code> as shown in my example.<br>
<code>model.apply</code> will recursively pass each module (and submodule, …) to the passed function.</p>

saba: <p>Sth Like that?</p>
<p>````` <code>def weights_init(m):</code></p>
<p><code>         </code> <code>with torch.no_grad():</code></p>
<p><code>               </code> <code>if isinstance(m, nn.Conv3d):</code></p>
<p><code>                      </code> <code>torch.nn.init.normal_(m.weight)</code></p>
<p><code>                      </code> <code>torch.nn.init.normal_(m.bias)</code></p>
<p>model = ConvNetRedo1(32,64,(7,7,5),(5,5,3),500,100)</p>
<p><code>model.apply(weight_init)</code></p>
<p><code>model=model.cuda</code></p>
<p><code>for epoch in range(num_epochs):</code></p>
<p><code>      </code> <code>for i, data  in enumerate(trainloader,0):</code></p>
<p><code>             </code> <code>images, labels=data   </code></p>
<p><code>              </code> <code>optimizer.zero_grad()</code></p>
<p><code>               </code> outputs= model(images)````</p>

ptrblck: <p>Yes, that looks generally right.<br>
You should call <code>model = model.cuda()</code> as a method (with parentheses), but the initialization work flow looks correct.</p>
<p>PS: It looks like you are adding each line in the format ticks `. Just add ``` before and after the complete code block (or use the “Preformatted text” button. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

saba: <p>I really appreciate your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Excuse me what is (net.apply(weights_init)) in your code. I did not use it . is it correct? Or I missed it?</p>
<pre><code class="lang-auto">   with torch.no_grad():
       if isinstance(m, nn.Conv3d):
           torch.nn.init.normal_(m.weight)
           torch.nn.init.normal_(m.bias)

net.apply(weights_init)</code></pre>