Forward function convert negative numbers to zero

Abdelpakey: <p>Hi,</p>
<p>I have this snippet code, I have divided Alexnet to 3 sub-modules because I want to take the output of each sub-module to feed it to another network later.</p>
<p>class alex (nn.Module):<br>
def <strong>init</strong>(self):<br>
super(alex,self).<strong>init</strong>()<br>
self.net= models.alexnet(pretrained= True)<br>
for param in self.net.parameters():<br>
param.requires_grad=False<br>
self.feat_list=list(self.net.features.children())<br>
self.feat_model= nn.Sequential(*self.feat_list)</p>
<pre><code>        self.c1= list(self.net.classifier.children())[:-2]
        self.c1.pop(2)
        self.c1.insert(2, nn.ReLU(0.2))            
        self.sub_cl1=nn.Sequential(*self.c1)            
        self.c2= list(self.net.classifier.children())[5:7]
        self.sub_cl2=nn.Sequential(*self.c2)  
def forward(self,x):
          x_feat= self.feat_model(x)
          x_feat=self.net.avgpool(x_feat)
          x_feat=x_feat.view(-1,x_feat.size(1)*x_feat.size(2)*x_feat.size(3))
</code></pre>
<p>===&gt;    x_sub_classifier1= self.sub_cl1(x_feat)<br>
===&gt;    model=self.sub_cl2(x_sub_classifier1)<br>
return model, x_feat, x_sub_classifier1</p>
<p>The problem is during the runtime I debug the code and I can see the output of these variables:<br>
x_feat<br>
==&gt;x_sub_classifier1<br>
model<br>
everything seems fine but step ahead from x_sub_classifier1 to model all negative values in x_sub_classifier1 become zero (note that I take the output after the Linear layer so there is no Relu function after the linear function to make the negative values Zeros) and the sub_cl1 has this sub-architecture</p>
<p>(sub_cl1): Sequential(<br>
(0): Dropout(p=0.5, inplace=False)<br>
(1): Linear(in_features=9216, out_features=4096, bias=True)<br>
(2): ReLU(inplace=True)<br>
(3): Dropout(p=0.5, inplace=False)<br>
(4): Linear(in_features=4096, out_features=4096, bias=True)<br>
)</p>
<p>Am I missing something or there is something I don’t understand?</p>

ptrblck: <p><code>self.sub_cl2</code> uses an inplace <code>ReLU</code> as the first layer, which will (as the name suggests) apply the relu inplace on the input tensor, which will thus apply it on <code>x_sub_classifier1</code>.<br>
You could set the <code>inplace</code> argument to <code>False</code> or alternatively clone <code>x_sub_classifier1</code> before passing it to <code>self.sub_cl2</code>.</p>
<p>PS: Unrelated to your current issue, but you should use <code>nn.ModuleList</code> instead of Python <code>lists</code>, as the former approach will make sure to properly register all parameters.</p>

Abdelpakey: <p>Thanks for your reply<br>
Based on what you have pointed out, is that the right way to re-write the previous code?</p>
<p>class alex (nn.Module):</p>
<pre><code>def __init__(self):
        super(alex,self).__init__()
        self.net= models.alexnet(pretrained= True)
        self.feat_model = []
        self.sub_cl1 = []
        self.sub_cl2 = []
        for param in self.net.parameters():
            param.requires_grad=False
        self.feat_model.append(self.net.features)
        self.sub_cl1.append(self.net.classifier[:-2])
        self.sub_cl2.append(self.net.classifier[5:7])
        self.sub_cl2[0][0]=nn.ReLU()

def forward(self,x):
    
          for layer in self.feat_model:
            feature = layer(x)
          x_feat=self.net.avgpool(feature)
          x_feat=x_feat.view(-1,x_feat.size(1)*x_feat.size(2)*x_feat.size(3))                
          for subm1 in self.sub_cl1:
            x_sub_classifier1 = subm1(x_feat)                
          for subm2 in self.sub_cl2:
            model= subm2(x_sub_classifier1)        
          return feature,  x_sub_classifier1,model</code></pre>

Abdelpakey: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> is the code above correct?</p>

ptrblck: <p>The code looks alright for changing the inplace behavior.<br>
Do you still see the issue using this model?</p>
<p>Unrelated to the current issue, but you should use <code>nn.ModuleList</code> instead of Python lists, as the former approach will make sure to properly register all modules inside the model.</p>

Abdelpakey: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
<p>I’m no longer seeing the problem related to Relu. I used nn.ModuleList and yes it registered all layers parameters.</p>
<p>Thanks for your helping!</p>