Training time when using Dropout with p=0.0

username: vdw
cooked: <p>I want to keep my model configurable including setting the Dropout probability to different values. The easiest way would be to create all my Dropout layers something like</p>
<pre><code class="lang-auto">self __init__(self, ..., drobout_prob=0.0, ...):
    ...
    self.dropout1 = nn.Dropout(p=dropout_prob)
    ...

self forward(X):
    ...
    out = self.dropout1(out)
    ...
</code></pre>
<p>I have no doubt this works quite fine. I only wonder if I sacrifice any noteworthy performance if my dropout probability is 0.0, making all Dropout layers essentially identity functions. In principle, I could do something like</p>
<pre><code class="lang-auto">self forward(X):
    ...
    if self.dropout_prob is not None and self.dropout_prob &gt; 0.0:
        out = self.dropout1(out)
    ...
</code></pre>
<p>Would this have any measurable advantage in practice?</p>

username: jondapper
cooked: <p>Internally, dropout with p=0 behaves as identity:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/1d705b4b075e32540293d1717b582442a66ffcce/aten/src/ATen/native/Dropout.cpp#L44">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/1d705b4b075e32540293d1717b582442a66ffcce/aten/src/ATen/native/Dropout.cpp#L44" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/1d705b4b075e32540293d1717b582442a66ffcce/aten/src/ATen/native/Dropout.cpp#L44" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/1d705b4b075e32540293d1717b582442a66ffcce/aten/src/ATen/native/Dropout.cpp#L44</a></h4>



    <pre class="onebox"><code class="lang-cpp">
      <ol class="start lines" start="34" style="counter-reset: li-counter 33 ;">
          <li>
          </li>
<li>template&lt;bool inplace&gt;</li>
          <li>Tensor multiply(const Tensor&amp; input, const Tensor&amp; noise) {</li>
          <li>  static_assert(!inplace, "Wrong multiply overload triggered in Dropout.cpp");</li>
          <li>  return input.mul(noise);</li>
          <li>}</li>
          <li>
          </li>
<li>template&lt;bool feature_dropout, bool alpha_dropout, bool inplace, typename T&gt;</li>
          <li>Ctype&lt;inplace&gt; _dropout_impl(T&amp; input, double p, bool train) {</li>
          <li>  TORCH_CHECK(p &gt;= 0 &amp;&amp; p &lt;= 1, "dropout probability has to be between 0 and 1, but got ", p);</li>
          <li class="selected">  if (p == 0 || !train || input.numel() == 0) {</li>
          <li>    return input;</li>
          <li>  }</li>
          <li>
          </li>
<li>  if (p == 1) {</li>
          <li>    return multiply&lt;inplace&gt;(input, at::zeros({}, input.options()));</li>
          <li>  }</li>
          <li>
          </li>
<li>  at::Tensor b; // used for alpha_dropout only</li>
          <li>  auto noise = feature_dropout ? make_feature_noise(input) : at::empty_like(input);</li>
          <li>  noise.bernoulli_(1 - p);</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
so there should be need for the if statement.</p>