How to ensure the deterministic batch data when using RandomSampler and DataLoader

pyxiea: <p>The code below is how I define the dataloader</p>
<pre><code class="lang-auto">train_dataset = TweetDataset(
            tweet=train_df.text.values,
            sentiment=train_df.sentiment.values,
            selected_text=train_df.selected_text.values
        )
train_sampler = RandomSampler(train_dataset)
train_loader = DataLoader(train_dataset, sampler=train_sampler, batch_size=BATCH_SIZE, pin_memory=True)
</code></pre>
<p>Due to some reason, I have to define two model, like this:</p>
<pre><code class="lang-auto">model = TweetModel()
swa_model = TweetModel()
</code></pre>
<p>But I found that after each time of initiating a model, the dataloader defined above will be affected, for example, the first batch will be different.</p>
<p>As a result, even I set every random seed I can think of, the behavior of dataloader is “not deterministic”. I can confirm that if I always use one model(or always use two model), the data of every batch is deterministic, and my whole experiment is reproducible.</p>
<p>But here is the problem, I have to compare the performance between 1) using one model and 2) using two model. Due to the phenomenon above, they will give me different batch data, which will lead to different performance (~0.003, not very big, but very essential in a competition).</p>
<p>I know the impact happens in the <code>__init__</code> method of <code>TweetModel</code>, here is its definition:</p>
<pre><code class="lang-auto">from transformers import BertPreTrainedModel,RobertaModel

class TweetModel(BertPreTrainedModel):
    def __init__(self):
        super(TweetModel, self).__init__(config)
        self.bert = RobertaModel.from_pretrained(model_dir, config=config)
        # do something

    def forward(self, ids, mask, token_type_ids):
        #do something
</code></pre>
<p>After debug for a while, I notice that after the line <code>RobertaModel.from_pretrained(model_dir, config=config)</code>, the data of the fist batch of <code>train_loader</code> will be different. I check this by adding the following snippet before and after the line <code>self.bert =...</code> and compare the first batch data:</p>
<pre><code class="lang-auto">tmp=None
for d in train_loader:
    tmp=d
    break
# stop here and check the data of the first batch
</code></pre>
<p>I don’t know why loading  a pretrained model will affect <code>train_loader</code>, this is an unexpected behavior. Why would this happen and how to avoid it? Am I missing something? Any help will be highly appreciated.</p>
<p>PS: I noticed this issue in pytorch repo, <a href="https://github.com/pytorch/pytorch/issues/7209" rel="nofollow noopener">[Feature Request][PyTorch] Deterministic and predictable behaviour of batch sampler </a>, but it seems like the problem mentioned in the issue has been fixed and I don’t know whether it is related to my question.</p>

ptrblck: <p>Each call into the pseudo-random number generator will increase its state.<br>
Your model creation is most likely calling into a the pseudo-random number generator at some point, which will make all following (random) calls different.</p>
<p>A simple code snippet is here:</p>
<pre><code class="lang-python">torch.manual_seed(2809)

lin1 = nn.Linear(100, 100)
# Comment out lin2
lin2 = nn.Linear(100, 100) # another call to the PRNG
for _ in range(10):
    print(torch.randn(1))
</code></pre>
<p>Each run yields deterministic results for itself, but if you comment <code>lin2</code>, the PRNG will not be called for the model instantiation, and thus the following <code>torch.randn</code> calls will still yield deterministic results, but different values compared to the first run.</p>
<p>Have a look at the <a href="https://en.wikipedia.org/wiki/Pseudorandom_number_generator">Wikipedia article about PRNG</a> for more information.</p>
<p>If you need deterministic results for your workflow, I would recommend to initialize both models once and store their <code>state_dicts</code>.<br>
In the test scripts create the model, load their <code>state_dicts</code>, and set the manual seeds afterwards to reset the PRNG for the training.</p>
<p>Note, since the <code>forward</code> methods of both models might also contain calls to the PRNG, the next iteration might again not yield the same batch.<br>
If you see this behavior, you would have to make sure to use exactly the same call sequence and discard one model, if its output is not needed for the run.</p>

pyxiea: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for your reply!</p>