Using same dropout object for multiple drop-out layers?

Muppet: <p>Hi, can I use the same dropout object for multiple drop-out layers? And the same ReLU object? Or do these need to be created individually for each separate use in a layer?</p>
<p>e.g. this:</p>
<pre><code class="lang-auto">class Model(nn.Module):
  def __init__(self):
    super().__init__()
    self.dropout = nn.Dropout(0.5)
    self.relu = nn.ReLU()
    self.lin1 = nn.Linear(4096, 4096)
    self.lin2 = nn.Linear(4096, 4096)
    self.lin3 = nn.Linear(4096, 4096)
  def forward(self, x):
    output = self.dropout(self.relu(self.lin1(x)))
    output = self.dropout(self.relu(self.lin2(output)))
    output = self.dropout(self.relu(self.lin3(output)))
    ....
</code></pre>
<p>vs:</p>
<pre><code class="lang-auto">class Model(nn.Module):
  def __init__(self):
    super().__init__()
    self.dropout1 = nn.Dropout(0.5)
    self.dropout2 = nn.Dropout(0.5)
    self.dropout3 = nn.Dropout(0.5)
    self.relu1 = nn.ReLU()
    self.relu2 = nn.ReLU()
    self.relu3 = nn.ReLU()
    self.lin1 = nn.Linear(4096, 4096)
    self.lin2 = nn.Linear(4096, 4096)
    self.lin3 = nn.Linear(4096, 4096)
  def forward(self, x):
    output = self.dropout1(self.relu1(self.lin1(x)))
    output = self.dropout2(self.relu2(self.lin2(output)))
    output = self.dropout3(self.relu3(self.lin3(output)))
    ....
</code></pre>
<p>I am pretty sure it wouldn’t make a different for the ReLU, but I am not so convinced about the dropout?</p>

alex.veuthey: <p>You can definitely use the same ReLU activation, since it doesn’t have a specific state.</p>
<p>For dropout, I understand why it could not work, but the <code>nn.Dropout</code> module itself calls the functional API <code>F.dropout</code> at each <code>forward</code> call, so it would seem that each call randomizes the dropped weights, regardless of whether it’s several modules or just the one! See the source code <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/dropout.html" rel="nofollow noopener">here</a>.</p>

truGeneral7: <p>Hi, have you figured this out? I am having the same question.</p>

alex.veuthey: <p>Have a look <a href="https://discuss.pytorch.org/t/using-same-dropout-object-for-multiple-drop-out-layers/39027/2">here</a>.</p>

truGeneral7: <p>I actually have seen your reply but not sure if that is what the question asked. But thanks for following up.</p>

tom: <p>Excuse me for jumping into the conversation, but so this was one of the bits of style advice in a <a href="https://twitter.com/ThomasViehmann/status/1282555652112621568" rel="nofollow noopener">tutorial I did earlier this week</a>.</p>
<p>While it would technically work for vanilla PyTorch use, I would consider it bad advice to re-use layers. This includes ReLU and Dropout.</p>
<p>My style advice is to use the functional interface when you don’t want state, and instantiate an one object per use-case for if you do.</p>
<p>The reason for this is that it causes more confusion than benefits.</p>
<ul>
<li>It looks funny in printing.</li>
<li>It may screw up other analysis tools.</li>
<li>When you do advanced things, e.g. quantization, suddenly ReLU <em>is</em> stateful because it captures data for quantization.</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

truGeneral7: <p>Thank you Thomas, for your answer, I think that is what I was looking for. Also, I usually directly call ReLUs in forward without declaring them in init but wasn’t sure if that is the best practice.</p>

mathematics: <p>From my personal view,<br>
Using different dropouts would be better.<br>
Because using same dropout could cause to deactivate same connected neurons in different layers.<br>
Different dropouts would randomly deactivate mitigating overfitting.</p>

tom: <aside class="quote no-group" data-username="mathematics" data-post="8" data-topic="39027">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mathematics/48/40082_2.png" class="avatar"> mathematics:</div>
<blockquote>
<p>Because using same dropout could cause to deactivate same connected neurons in different layers.<br>
Different dropouts would randomly deactivate mitigating overfitting.</p>
</blockquote>
</aside>
<p>Did you try this, though?</p>
<p>I would consider it is a red herring as calling into the same dropout module will still give different random samples.<br>
Not re-using a single module is really a matter of cleanliness in expressing the structure as code.</p>
<p>Best regards</p>
<p>Thomas</p>

Aris_L: <p>After some test, different <code>F.dropout2d</code> calls seem to deactivate different part of the tensor.</p>

nikita_gordia: <p>Thx for you explanation, as I have understood during the inference of quantised model your ReLU works differently as if it was training (it learns params that fix quantization error during training). But it is not that clear about analysis tools, could you pls give an example of some case or tool?</p>