Should one set model.eval() when getting outputs in a training epoch

username: kpatil
cooked: <p>When using randomized/batch-dependent layers like dropout or BN is is necessary to set <code>model.eval()</code> before getting the outputs for a batch within a training epoch?</p>
<p>The question also appears on stackoverflow: <a href="https://stackoverflow.com/questions/63167099/pytorch-training-with-dropout-and-or-batch-normalization" rel="nofollow noopener">https://stackoverflow.com/questions/63167099/pytorch-training-with-dropout-and-or-batch-normalization</a></p>

username: tom
cooked: <p>I’d say it is a clear: <strong>No.</strong> You would keep the model in <em>training</em> state (<code>model.train()</code>) while training and only set it to <code>model.eval()</code> during evaluation (e.g. validation interleaved with training epochs, inference proper). Don’t for get to set it back to training state for the next epoch if you do.</p>
<p>I must admit I have trouble to see why the person helping you on stackoverflow would come to a different conclusion.</p>
<p>The other way round it would make more sense btw., for things like test time dropout, you would actually avoid setting the dropout layers to evaluation mode.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>P.S.: I would advise against crossposting. If you get the same answer, it’ll be wasted time for people answering with no benefit to you. If you get different ones, you need to make up your mind which would have been the better forum to ask in anyways and it’ll be wasted time for people answering.</p>

username: kpatil
cooked: <p>Thanks for quick reply.<br>
I do not get the point why a dropout layer should not to be set t eval mode. If it is not set to eval mode then the output for the same example can be different for different runs.</p>
<p>PS: as for cross-posting I think it’s not necessarily bad as the user base for different forums can be quite different. And its not necessarily my question on other forums.</p>

username: tom
cooked: <aside class="quote no-group" data-username="kpatil" data-post="3" data-topic="91069">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kpatil/48/27297_2.png" class="avatar"> kpatil:</div>
<blockquote>
<p>PS: as for cross-posting I think it’s not necessarily bad as the user base for different forums can be quite different. And its not necessarily my question on other forums.</p>
</blockquote>
</aside>
<p>I can’t help but feel it wastes my time. But of course, I can just not answer questions, too.</p>

username: Nikronic
cooked: <aside class="quote no-group" data-username="kpatil" data-post="3" data-topic="91069">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kpatil/48/27297_2.png" class="avatar"> kpatil:</div>
<blockquote>
<p>I do not get the point why a dropout layer should not to be set t eval mode. If it is not set to eval mode then the output for the same example can be different for different runs.</p>
</blockquote>
</aside>
<p>I think you need to study the reason for introducing dropout/bn in NNs. Their job is to modify outputs in each iteration. Then if we disable dropouts before feedforward, why don’t we just remove them?</p><aside class="onebox wikipedia" data-onebox-src="https://en.wikipedia.org/wiki/Dilution_(neural_networks)">
  <header class="source">

      <a href="https://en.wikipedia.org/wiki/Dilution_(neural_networks)" target="_blank" rel="noopener nofollow ugc">en.wikipedia.org</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://en.wikipedia.org/wiki/Dilution_(neural_networks)" target="_blank" rel="noopener nofollow ugc">Dilution (neural networks)</a></h3>

<p>Dilution and dropout (also called DropConnect) are regularization techniques for reducing overfitting in artificial neural networks by preventing complex co-adaptations on training data. They are an efficient way of performing model averaging with neural networks. Dilution refers to thinning weights, while dropout refers to randomly "dropping out", or omitting, units (both hidden and visible) during the training process of a neural network. Both trigger the same type of regulari Dilution is usual...</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>