reproduciblity compared with vanilla pytorch
🐛 Bug
I am trying to compare a vanilla pytorch code which I refactored as a part of my learning pytorch-lightning but I see that the training_step is iterating through a different order of my dataset inspite of me setting seed_everything and deterministic=True and benchmark=False
Please reproduce using [the BoringModel and post here]
I have this as the base, common for both vanilla mode and pytorch-lightning mode:
        train_loader = torch.utils.data.DataLoader(\
        MyDataSet(cfg=cfg, mode="train"), \
                            batch_size=16, \
                            shuffle=True)

In lightning I do:
    def training_step(self, batch, batch_idx):
        x, y, paths = batch
        
        if Pathlib("check_input_path_pl").is_file():
            append_write = 'a+' # append if already exists
        else:
            append_write = 'w' # make a new file if not
        with open("check_input_path_pl", append_write) as f:
            f.write(str(batch_idx) + " ==> " +str(paths[0]) + "\n")

        y_hat = self(x)
        loss = self.loss_func(y_hat, y)
       return loss

in vanilla pytorch mode I do:
    for batch_idx, (data, target, paths) in enumerate(train_loader):
        data, target = data.to(device), target.to(device)
        if Pathlib("check_input_path").is_file():
            append_write = 'a+' # append if already exists
        else:
            append_write = 'w' # make a new file if not
        with open("check_input_path", append_write) as f:
            f.write(str(batch_idx) + " ==> " +str(paths[0]) + "\n")

but when I compare check_input_path  with check_input_path_pl I see that after the first epoch, the order of the input file paths is different :(
I do:
seed_everything(111)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

in both vanilla and lightning mode.
Expected behavior
both should yield same order. To make a simple check, I iterate through the train_loader as soon as I create it in both pytl and pyt like so:
        for epoch in range(0, 2):
            for batch_idx, (data, target, paths) in enumerate(train_loader):
                if Pathlib("paths2load").is_file():
                    append_write = 'a+' # append if already exists
                else:
                    append_write = 'w' # make a new file if not
                with open("paths2load", append_write) as f:
                    f.write(str(batch_idx) + " ==> " +str(paths[0]) + "\n")

and in both pyt and pytl the files show the same order. yet, when training, they load in different order. What could the reason be?
Environment
pytorch-lightning==1.0.6
torch==1.6.0
torchvision==0.7.0
Debian Linux
PS: both pyt and pytl are running with the same virtualenv