Selectively applying random seed to different parts of the model

username: Vignesh_Baskaran
cooked: <p>I am trying to study the effect of random seeds on the dropout and linear initialization of the model. Here is a snippet of my model</p>
<pre><code class="lang-python">class PoolerClassifier(nn.Module):
    def __init__(self, transformer: nn.Module, transformer_dropout: float, linear_dropout: float,
                 input_size: int, num_classes: int, seed: int):
        super(PoolerClassifier, self).__init__()
        self.transformer = transformer(transformer_dropout)  # Transformer dropout here
        self.linear_dropout = nn.Dropout(p=linear_dropout)  # Linear dropout here
        self.linear = nn.Linear(input_size, num_classes)  # Linear initialization here

    def forward(self, inputs):
        transformer_outputs = self.transformer(**inputs)
        vectors = self.linear_dropout(transformer_outputs.pooler_output)
        logits = self.linear(vectors)
        return logits
</code></pre>
<p>I have three questions:<br>
The first one is: Ideally I would like to assign different random seeds to:</p>
<ol>
<li>transformer dropout: random_seed(1)</li>
<li>linear dropout: random_seed(2)</li>
<li>linear initialization: random_seed(3)</li>
</ol>
<p>My second question is: How can I selectively apply random seed to any of them. To elaborate, I would like to do the following:</p>
<ol>
<li>transformer dropout - Set random seed - So deterministic behaviour</li>
<li>linear dropout - No random seed - So random behaviour</li>
<li>linear initialization - Set random seed - So deterministic behaviour</li>
<li>
<strong>Very importantly</strong>: Remove random seed so everything else following the model has non-deterministic behaviour</li>
</ol>
<p>My third question is: Can I do something like this: <code>nn.Dropout(random_state=12345)</code>?</p>
<p>Thank you very much for your help.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Vignesh_Baskaran" data-post="1" data-topic="131315">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vignesh_baskaran/48/39656_2.png" class="avatar"> Vignesh_Baskaran:</div>
<blockquote>
<p>The first one is: Ideally I would like to assign different random seeds to:</p>
<ol>
<li>transformer dropout: random_seed(1)</li>
<li>linear dropout: random_seed(2)</li>
<li>linear initialization: random_seed(3)</li>
</ol>
</blockquote>
</aside>
<p>You could set the seed using <code>torch.manual_seed</code> before running these operations. Note however that seeding the code before applying a random operation (such as using dropout) would use the same mask for the same input shape, so this operation isn’t “random” anymore.</p>
<aside class="quote no-group" data-username="Vignesh_Baskaran" data-post="1" data-topic="131315">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vignesh_baskaran/48/39656_2.png" class="avatar"> Vignesh_Baskaran:</div>
<blockquote>
<p>My second question is: How can I selectively apply random seed to any of them. To elaborate, I would like to do the following:</p>
<ol>
<li>transformer dropout - Set random seed - So deterministic behaviour</li>
<li>linear dropout - No random seed - So random behaviour</li>
<li>linear initialization - Set random seed - So deterministic behaviour</li>
<li><strong>Very importantly</strong>: Remove random seed so everything else following the model has non-deterministic behaviour</li>
</ol>
</blockquote>
</aside>
<p>You cannot switch between deterministic and random behavior easily, since all operations using the pseudorandom number generator would use the already set seed.<br>
E.g. if you set the seed in operation 1 (transformer dropout), the same drop mask will be samples, but also the following operations using the PRNG would use the same “random” numbers unless you reseed the code to a new (random) seed.</p>

username: Vignesh_Baskaran
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>. I am going to try out your suggestions and write back to you. Thank you</p>