Doubt in the code of Faster RCNN implemenation on GitHub

username: jitesh
cooked: <p>I am trying to implement Faster RCNN for Object Detection. I am following <a href="https://github.com/jwyang/faster-rcnn.pytorch" rel="nofollow noopener">this</a> particular GitHub repo for implementation. However, I have a doubt from <a href="https://github.com/tdchaitanya/MMTOD/blob/master/lib/model/faster_rcnn/resnet.py#L238" rel="nofollow noopener">this</a> particular line in the code and just want to clarify if I have understood things correctly. There is block of code following this line where a few layers have been fixed by setting <code>requires_grad</code> to <code>False</code>. However, in <a href="https://github.com/tdchaitanya/MMTOD/blob/3a438121c663c2af00da46fed3b467c2012c61ed/lib/model/faster_rcnn/resnet.py#L274" rel="nofollow noopener">this</a> particular line in the train function, the base feature extractor is set to <code>eval()</code> but then again the following 2 lines train some specific layers.</p>
<p>I have 2 main doubts,</p>
<ol>
<li>Why is <code>requires_grad</code> set to <code>False</code> in the preceding few lines if it is anyway going to be used in evaluation mode?</li>
<li>What effect do the 2 lines below the <code>self.FRCNN_base.eval()</code> do in terms of training in the <code>def train()</code> function?</li>
</ol>
<p>Any help would be appreciated. Thanks</p>

username: simaiden
cooked: <p>It seems that  is using a pretrained model to do transfer learning/fine tuning, so you don’t want to train early layers of the net (in this case the R-CNN) but classification/regression layers.</p>
<ol>
<li>
<p><code>requires_grad = False</code> means you are freezing those layers in training, and model.eval() is to prepare the batchnorm/dropout layers to act in evaluation mode, so isn’t the same.</p>
</li>
<li>
<p>I guess it’s to put classification and regression layers into training mode.</p>
</li>
</ol>

username: jitesh
cooked: <p><a class="mention" href="/u/simaiden">@simaiden</a>, thanks for the reply. But If you might have observed in the lines 266 and 283, there is a separate function which is being called to set the batch norm layers in eval mode. Why is this being done if the whole model is anyway being set to eval mode ?</p>

username: jitesh
cooked: <p>Also, are the other layers such as <code>self.RCNN_cls_score</code>  and <code>self.RCNN_bbox_pred</code> in lines 243 and 245 respectively being trained? I mean, they  ideally should be, because otherwise, I don’t think we would have good results(please correct me if I am wrong). However, I got this doubt because these layers are not explicitly set to <code>training mode</code> in the <code>def train()</code> function.</p>

username: ptrblck
cooked: <p>The <code>RCNN_cls_score</code> and <code>RCNN_bbox_pred</code> should be trained, as I cannot see where their <code>requires_grad</code> attribute is manipulated.</p>
<p>As <a class="mention" href="/u/simaiden">@simaiden</a> said, the <code>eval()</code> and <code>train()</code> calls only change the behavior of batchnorm and dropout layers. They are not freezing the layers.</p>

username: jitesh
cooked: <p>Okay, so this is what I feel I have understood from the above discussion (please correct me if I am wrong)</p>
<ol>
<li>
<p>The only difference between <code>eval()</code> and <code>train()</code> is the effect they have on the <code>BatchNorm</code> and <code>Dropout</code> layers. Otherwise, if the <code>requires_grad()</code> attribute of the layers is set to <code>True</code>, those layers will be trained and backprop will still happen for those layers irrespective of whether they are set to <code>eval()</code> or <code>train()</code></p>
</li>
<li>
<p>Now coming to the code in the repo, in the <code>def train()</code> function, since the <code>self.RCNN_base</code> is anyway set to <code>eval()</code>, am I correct in assuming that the following line of code <code>self.RCNN_base.apply(set_bn_eval)</code> in line 283 is redundant  since the <code>Batchnorm</code> Layers are already set to eval inside the <code>if</code> condition at line 272?</p>
</li>
</ol>

username: ptrblck
cooked: <ol>
<li>
<p>That’s correct. Note that other (custom) layers might also change their behavior via <code>train()/eval()</code>, if they use the <code>self.training</code> argument internally.</p>
</li>
<li>
<p>Not necessarily. L274 sets the base to <code>eval()</code>, while L275+ sets the 5th and 6th module back to train. Afterwards all batchnorm layers are reset to back to <code>eval()</code> (in particular, if <code>RCNN_base[5]</code> or <code>RCNN_base[6]</code> contain batchnorm layers).</p>
</li>
</ol>

username: jitesh
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks for the clarification. How about if <code>requires_grad</code> is set to <code>False</code> ? Will setting <code>model.train()</code> or <code>model.eval()</code> (in terms of Batchnorm and Dropout Layers) make a difference for those layers whose parameters have <code>requires_grad</code> set to <code>False</code> ?</p>
<p>And could you also mention if there are any other layers apart from<code>BatchNorm</code> and <code>Dropout</code> whose behaviour changes upon using <code>.train()</code> and <code>.eval()</code> ? Thanks!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="jitesh" data-post="8" data-topic="74243">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jitesh/48/11907_2.png" class="avatar"> jitesh:</div>
<blockquote>
<p>Will setting <code>model.train()</code> or <code>model.eval()</code> (in terms of Batchnorm and Dropout Layers) make a difference for those layers whose parameters have <code>requires_grad</code> set to <code>False</code> ?</p>
</blockquote>
</aside>
<p>Yes, <code>eval()/train()</code> do not change anything about the gradients and if parameters require gradients, but change the behavior (code paths) of some layers.</p>
<aside class="quote no-group" data-username="jitesh" data-post="8" data-topic="74243">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jitesh/48/11907_2.png" class="avatar"> jitesh:</div>
<blockquote>
<p>And could you also mention if there are any other layers apart from <code>BatchNorm</code> and <code>Dropout</code> whose behaviour changes upon using <code>.train()</code> and <code>.eval()</code> ?</p>
</blockquote>
</aside>
<p>If I’m not mistaken, batchnorm and dropout layers are the only layers at the moment in the PyTorch core, which change the behavior.<br>
However, custom layers (e.g. if you are using a specific repository from a 3rd party) can easily use the <code>self.training</code> flag to change the layer.</p>

username: jitesh
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="9" data-topic="74243">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Yes, <code>eval()/train()</code> do not change anything about the gradients and if parameters require gradients, but change the behavior (code paths) of some layers.</p>
</blockquote>
</aside>
<p>You mean to say eval() and train() don’t change anything if the parameters have <code>requires_grad</code> set to <code>False</code> right ? And could you also give examples of where the behaviour might change (as you mentioned in the answer)?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="jitesh" data-post="10" data-topic="74243">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jitesh/48/11907_2.png" class="avatar"> jitesh:</div>
<blockquote>
<p>You mean to say eval() and train() don’t change anything if the parameters have <code>requires_grad</code> set to <code>False</code> right ?</p>
</blockquote>
</aside>
<p>No. <code>train()</code> and <code>eval()</code> are completely independent of the <code>requires_grad</code> attribute of any parameters.<br>
These methods change the <code>self.training</code> attribute of the model to <code>True</code> or <code>False</code>, which is then used to trigger different code paths.</p>
<aside class="quote no-group" data-username="jitesh" data-post="10" data-topic="74243">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jitesh/48/11907_2.png" class="avatar"> jitesh:</div>
<blockquote>
<p>And could you also give examples of where the behaviour might change (as you mentioned in the answer)</p>
</blockquote>
</aside>
<p>Dropout will be disabled when <code>eval()</code> is used and batchnorm layers will use their running stats to normalize the input activation instead of the batch stats.</p>

username: jitesh
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, Thank you for the clarification, I am really sorry to keep troubling you with a barrage of questions <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:">. I actually have a few more doubts about <code>BatchNorm</code> layers in PyTorch and its behavior. So can we have that discussion in this post itself or is there any specific post (I have found quite a few about BatchNorm!) I can post my questions on?<br>
Or should I open a new topic altogether? Thanks!</p>

username: ptrblck
cooked: <p>If it’s unrelated to the original question, please open a new topic and I’ll have a look. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: jitesh
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, I have made a new topic regarding my issue. Here is the <a href="https://discuss.pytorch.org/t/using-batchnorm-gives-a-significantly-low-accuracy/74525">link</a></p>
<p>Thanks!</p>