auto_lr_find with ddp and slurm
🐛 Bug

If auto_lr_find is used in ddp mode with num_nodes > 1 and slurm you will get an error that the checkpoint file was not found and therefore could not be deleted:
  File "MyModel.py", line 63, in <module>
    trainer.tune(model, dm)
  File "/gpfs/home/username/miniconda3/envs/pytorch/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 804, in tune
    self.tuner.tune(model, train_dataloader, val_dataloaders, datamodule)
  File "/gpfs/home/username/miniconda3/envs/pytorch/lib/python3.8/site-packages/pytorch_lightning/tuner/tuning.py", line 54, in tune
    self.internal_find_lr(self.trainer, model)
  File "/gpfs/home/username/miniconda3/envs/pytorch/lib/python3.8/site-packages/pytorch_lightning/tuner/tuning.py", line 131, in internal_find_lr
    return _run_lr_finder_internally(trainer, model)
  File "/gpfs/home/username/miniconda3/envs/pytorch/lib/python3.8/site-packages/pytorch_lightning/tuner/lr_finder.py", line 43, in _run_lr_finder_internally
    lr_finder = lr_find(trainer, model)
  File "/gpfs/home/username/miniconda3/envs/pytorch/lib/python3.8/site-packages/pytorch_lightning/tuner/lr_finder.py", line 188, in lr_find
    os.remove(save_path)
FileNotFoundError: [Errno 2] No such file or directory: '/gpfs/home/username/my_project/lr_find_temp.ckpt'
To Reproduce

Unfortunately the BoringModel doesn't use a LightningDataModule and therefore trainer.tune(model, dataloader) would not work. That's why I created a minimal working example, which uses a LightningDataModule:
import pytorch_lightning as pl
import torch.nn as nn
import torch
import torch.nn.functional as F
from torchvision import transforms
from torch.utils.data import DataLoader, Dataset
import numpy as np


class MyDataset(Dataset):
    def __init__(self, latent_dim=10000, n_samples=10000):
        self.data = np.random.random([n_samples, latent_dim])

    def __getitem__(self, index):
        return torch.from_numpy(self.data[index]).float()

    def __len__(self):
        return len(self.data)


class MyDataModule(pl.LightningDataModule):
    def __init__(self, batch_size=8):
        super().__init__()
        self.batch_size = batch_size

    def setup(self, stage):
        self.train_dataset = MyDataset()

    def train_dataloader(self):
        return DataLoader(self.train_dataset, batch_size=self.batch_size)


class MyTestModel(pl.LightningModule):
    def __init__(self, lr=1e-3, batch_size=8, **kwargs):
        super().__init__()
        self.learning_rate = lr
        self.save_hyperparameters()
        width = 10000
        self.linear1 = nn.Linear(10000, width)
        self.linear2 = nn.Linear(width, 10000)

    def configure_optimizers(self):
        return torch.optim.AdamW(self.parameters(), lr=self.learning_rate)

    def forward(self, x):
        x = self.linear1(x)
        x = self.linear2(x)
        return x

    def training_step(self, batch, batch_idx):
        x = batch
        y_hat = self(x)
        loss = F.mse_loss(y_hat, x)
        return loss


if __name__ == "__main__":
    dm = MyDataModule()
    model = MyTestModel()

    trainer = pl.Trainer(auto_lr_find=True, gpus=1)
    trainer.tune(model, dm)
    trainer.fit(model, dm)
The slurm file would be:
#!/bin/bash

# SLURM SUBMIT SCRIPT
#SBATCH --job-name=Test-DDP
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1
#SBATCH --time=08:00:00
#SBATCH --account=username
#SBATCH --partition=pGPU
#SBATCH --exclusive


module load compilers/cuda/10.1
export CUDA_VISIBLE_DEVICES="0"
nvidia-smi
srun /gpfs/home/username/miniconda3/envs/pytorch/bin/python MyModel.py \
      --gpus 1 --num_nodes 2 --distributed_backend "ddp"

Environment

PyTorch Version (e.g., 1.0): 1.0.2
OS (e.g., Linux): Linux
How you installed PyTorch (conda, pip, source): conda
Build command you used (if compiling from source): -
Python version: 3.8
CUDA/cuDNN version: 10.1
GPU models and configuration: Many nodes with Tesla-V100, each node having exactly one.
Any other relevant information: Slurm for scheduling jobs.

Additional context

I think the problem is that each node tries to delete the checkpoint file. The first node succeeds with that, but when the
second node tries the file is not there anymore.