question: How to get the perfect reproducibility
context: Hi, I'm currently trying to finetune a pretrained BERT model for intent classification using Huggingface's Transformers library and Pytorch Lightning.
  The structure is simple where a linear classifier is simply put on the BERT encoder.
  I want to get the same result at the same seed setting, but although the whole setting including the seed is identical, the result changes.
  I thought if I pre-fix the seed using seed_everything and set the flag workers=True, I can get the exact same result, but I don't know what the problem is.
  The fun fact is that all executions save the exact same best checkpoint with identical valid accuracy and actually the flow of the training seems also the same.
  But after executing the test, the results are not same.
  The main codes are as follows.
  from transformers import BertConfig, BertTokenizer, BertModel
  from pytorch_lightning import Trainer, seed_everything
  from pytorch_lightning.callbacks import ModelCheckpoint
  
  def run(args):
      # For directory setting
      ...
  
      # Tokenizer & Model => This model is a pre-trained encoder, so I think there is no need to fix a random seed.
      config = BertConfig.from_pretrained('bert-base-uncased')
      tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
      model = BertModel.from_pretrained('bert-base-uncased')
  
      ...
      
      print("Loading datasets...")
      # For data loading
      train_set = Dataset(...)
      valid_set = Dataset(...)
      test_set = Dataset(...)
  
      total_train_steps = int(len(train_set) / batch_size * num_epochs)
      warmup_steps = int(total_train_steps * warmup_prop)
      
      # Random seed fixing for intent classification layer
      seed_everything(0, workers=True)
      
      # Lightning Module setting  
      module = TrainModule(model)
  
      # Dataloaders
      ppd = PadCollate(...)
      
      # Reset random seed for data shuffle
      seed_everything(0, workers=True)
  
      train_loader = DataLoader(train_set, collate_fn=ppd.pad_collate, batch_size=batch_size, shuffle=True, num_workers=num_workers, pin_memory=True)
      valid_loader = DataLoader(valid_set, collate_fn=ppd.pad_collate, batch_size=batch_size, num_workers=num_workers, pin_memory=True)
      test_loader = DataLoader(test_set, collate_fn=ppd.pad_collate, batch_size=batch_size, num_workers=num_workers, pin_memory=True)
  
      print("Setting pytorch lightning callback & trainer...")
      # Model checkpoint callback
      filename = "best_ckpt_{epoch}_{train_all_acc:.4f}_{valid_all_acc:.4f}"
      monitor = "valid_all_acc"
      
      checkpoint_callback = ModelCheckpoint(
          dirpath=save_dir,
          filename=filename,
          verbose=True,
          monitor=monitor,
          mode='max',
          every_n_val_epochs=1,
      )
      
      # Trainer setting
      trainer = Trainer(
          check_val_every_n_epoch=1,
          gpus=gpu,
          auto_select_gpus=True,
          num_nodes=num_nodes,
          max_epochs=num_epochs,
          gradient_clip_val=max_grad_norm,
          num_sanity_val_steps=0,
          deterministic=True,
          callbacks=[checkpoint_callback]
      )
      
      print("Train starts.")
      trainer.fit(model=module, train_dataloader=train_loader, val_dataloaders=valid_loader)
      print("Training done.")
      
      print("Test starts.")
      trainer.test(model=module, test_dataloaders=test_loader, ckpt_path='best')
      
      print("GOOD BYE.")
  Also, TrainModule is designed like below.
  from torch import nn as nn
  import pytorch_lightning as pl
  
  class TrainModule(pl.LightningModule):
      def __init__(self, args, encoder):
          super().__init__()
          
          self.args = args
          self.save_hyperparameters(args)
          
          self.encoder = encoder
          self.output_layer = IntentDetection(args)
          self.output_layer.init_params()
          
          self.loss_func = nn.CrossEntropyLoss()
          
      def forward(self, input_ids, padding_masks=None):  # input_ids: (B, L), padding_masks: (B, L)
          hidden_states = self.encoder(input_ids=input_ids, attention_mask=padding_masks)[0]  # (B, L, d_h)
              
          return self.output_layer(hidden_states[:, 0])  # (B, L, C) or  (B, C)
      
      def training_step(self, batch, batch_idx): 
          ...
  
  class IntentDetection(nn.Module):
      def __init__(self, args):
          super(IntentDetection, self).__init__()
          
          self.hidden_size = args.hidden_size
          self.num_classes = args.num_classes
          
          self.linear = nn.Linear(self.hidden_size, self.num_classes)
          
      def forward(self, hiddens):
          # hiddens: (B, d_h)
          
          return self.linear(hiddens)  # (B, C)
      
      def init_params(self):
          nn.init.xavier_uniform_(self.linear.weight)  
  I also post the current training environment.
  
  OS: Ubuntu 18.04.5 LTS
  Python version: 3.8.5
  Pytorch version: 1.7.1+cu110
  Pytorch Lightning version: 1.3.0
  GPU: A100-SXM4-40GB (DGX)
  CUDA version: 11.0
  
  And I got 3 different results when I run the same codes 3 times.
  
  This is odd, since when I run the exact same code above in different environment, I could get the identical results from a same seed, even the number of workers in a dataloader is different.
  I also attach the environment which I was able to get the perfect reproducibility.
  
  OS: CentOs Linux release 7.9.2009 (Core)
  Python version: 3.7.4
  Pytorch version: 1.7.1+cu110
  Pytorch Lightning version: 1.3.0
  GPU: RTX 3090
  CUDA version: 11.2
  
  It will be really great if anyone can help me to solve this issue...
  Thank you very much.
answer: Can you try seeding again right before the trainer.test call?
  Not saying you should need to but to know if that makes any difference
  I think your best bet is to try to create a reproducible snippet. You can get started with the https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pl_examples/bug_report_model.py