Unable to make the reproducible result

loveis98: <p>This is my code to set the seed values right after the imports:</p>
<pre><code class="lang-auto">def seed_everything(seed):
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    np.random.seed(seed)  
    random.seed(seed) 
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True
    torch.use_deterministic_algorithms(True)
    os.environ['PYTHONHASHSEED'] = str(seed)
</code></pre>
<p>But I can’t achieve reproducible result. For <a href="https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html" rel="noopener nofollow ugc">example</a>:</p>
<pre><code class="lang-auto">import os
import pandas as pd
import numpy as np
import sys
import torch
import torch.nn as nn
import random 
import torch.nn.functional as F
import torch.optim as optim
print(torch.cuda.is_available())
os.environ["CUDA_VISIBLE_DEVICES"] = '1'
os.environ['CUBLAS_WORKSPACE_CONFIG']=':4096:8'
pd.set_option('display.max_columns', None)

# добавим корневую папку, в ней лежат все необходимые полезные функции для обработки данных
sys.path.append('./../../')
sys.path.append('./../')

def seed_everything(seed):
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    np.random.seed(seed)  
    random.seed(seed) 
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True
    torch.use_deterministic_algorithms(True)
    os.environ['PYTHONHASHSEED'] = str(seed)


seed_everything(7)

def prepare_sequence(seq, to_ix):
    idxs = [to_ix[w] for w in seq]
    return torch.tensor(idxs, dtype=torch.long)

training_data = [
    ("The dog ate the apple".split(), ["DET", "NN", "V", "DET", "NN"]),
    ("Everybody read that book".split(), ["NN", "V", "DET", "NN"])
]
word_to_ix = {}
for sent, tags in training_data:
    for word in sent:
        if word not in word_to_ix:  # word has not been assigned an index yet
            word_to_ix[word] = len(word_to_ix)  # Assign each word with a unique index
tag_to_ix = {"DET": 0, "NN": 1, "V": 2}  # Assign each tag with a unique index


EMBEDDING_DIM = 6
HIDDEN_DIM = 6

class LSTMTagger(nn.Module):

    def __init__(self, embedding_dim, hidden_dim, vocab_size, tagset_size):
        super(LSTMTagger, self).__init__()
        self.hidden_dim = hidden_dim

        self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim)
        self.hidden2tag = nn.Linear(hidden_dim, tagset_size)

    def forward(self, sentence):
        embeds = self.word_embeddings(sentence)
        lstm_out, _ = self.lstm(embeds.view(len(sentence), 1, -1))
        tag_space = self.hidden2tag(lstm_out.view(len(sentence), -1))
        tag_scores = F.log_softmax(tag_space, dim=1)
        return tag_scores
    
model = LSTMTagger(EMBEDDING_DIM, HIDDEN_DIM, len(word_to_ix), len(tag_to_ix)).to('cuda')
loss_function = nn.NLLLoss().to('cuda')
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

for epoch in range(10):  
    for sentence, tags in training_data:
        model.zero_grad()
        sentence_in = prepare_sequence(sentence, word_to_ix)
        targets = prepare_sequence(tags, tag_to_ix)

        tag_scores = model(sentence_in.to('cuda'))
        loss = loss_function(tag_scores, targets.to('cuda'))
        print(loss)
        loss.backward()
        optimizer.step()
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de7373bf4b8b87ace0bf1da6502e13a2d5e1420c.png" data-download-href="https://discuss.pytorch.org/uploads/default/de7373bf4b8b87ace0bf1da6502e13a2d5e1420c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de7373bf4b8b87ace0bf1da6502e13a2d5e1420c.png" alt="image" data-base62-sha1="vJTlURS9zSDEa6yVJVN7Lrd1Vak" width="568" height="500" data-dominant-color="E5E5E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">697×613 10.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But but for the next training of the same model:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/5/25afa6533c002b91723311c7e2b6072ede663213.png" data-download-href="https://discuss.pytorch.org/uploads/default/25afa6533c002b91723311c7e2b6072ede663213" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25afa6533c002b91723311c7e2b6072ede663213_2_501x500.png" alt="image" data-base62-sha1="5nnZDApO7Nnm8k3Tpvdhc3GIS3x" width="501" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25afa6533c002b91723311c7e2b6072ede663213_2_501x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25afa6533c002b91723311c7e2b6072ede663213_2_751x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25afa6533c002b91723311c7e2b6072ede663213_2_1002x1000.png 2x" data-dominant-color="EDEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1105×1102 138 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>That is, losses are not reproduced with the same seed</p>

JuanFMontesinos: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d50f3bafe943519141f48f8e9f9f901e794c360.png" data-download-href="https://discuss.pytorch.org/uploads/default/0d50f3bafe943519141f48f8e9f9f901e794c360" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d50f3bafe943519141f48f8e9f9f901e794c360.png" alt="image" data-base62-sha1="1TNE3IWSyspfT08C0XU9HRFsZO0" width="690" height="300" data-dominant-color="2D2D2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1198×522 26.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Actually I can mimic your first run with collab…<br>
It’s fully deterministic.</p>
<p>Are you rerunning the script?</p>